Oct 31 17:26:46.970643 ip-10-0-139-107 systemd[1]: Starting Kubernetes Kubelet... Oct 31 17:26:47.416603 ip-10-0-139-107 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 17:26:47.416603 ip-10-0-139-107 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 17:26:47.416603 ip-10-0-139-107 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 17:26:47.416603 ip-10-0-139-107 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 17:26:47.421978 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.421894 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:26:47.421978 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.421975 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:26:47.421978 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.421979 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:26:47.421978 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.421982 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:26:47.421978 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.421985 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.421988 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.421991 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.421995 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.421998 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422001 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422004 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422007 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422010 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422016 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422019 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422022 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422024 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422027 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422030 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422032 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422035 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422038 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422040 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422043 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:26:47.422138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422046 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422049 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422051 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422054 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422056 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422059 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422062 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422064 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422067 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422070 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422072 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422075 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422077 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422080 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422084 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422087 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422089 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422092 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422094 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422097 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:26:47.422614 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422099 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422102 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422105 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422107 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422110 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422112 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422115 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422117 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422120 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422122 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422125 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422128 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422131 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422133 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422137 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422140 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422142 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422145 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:26:47.423076 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422148 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422151 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422153 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422156 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422159 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422161 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422164 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422166 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422171 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422176 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422179 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422182 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422184 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422187 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422189 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422192 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422194 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422197 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422199 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422202 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:26:47.423492 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422205 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422208 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422210 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422213 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422215 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422217 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422220 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422223 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422225 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422228 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422230 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422233 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422236 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422238 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422241 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422243 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422246 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422262 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422267 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:26:47.423962 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422271 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422276 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422279 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422283 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.422287 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423111 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423118 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423121 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423123 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423126 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423129 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423134 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423139 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423142 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423145 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423148 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423150 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423153 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423156 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:26:47.424429 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423158 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423161 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423164 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423167 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423170 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423172 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423176 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423178 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423181 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423184 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423186 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423189 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423191 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423194 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423198 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423200 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423203 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423206 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423210 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:26:47.424880 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423213 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423215 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423218 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423220 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423224 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423228 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423231 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423234 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423236 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423239 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423242 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423245 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423247 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423266 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423269 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423272 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423274 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423277 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423280 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423284 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:26:47.425340 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423287 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423290 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423292 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423295 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423298 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423300 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423303 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423306 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423309 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423311 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423314 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423317 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423319 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423322 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423325 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423327 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423330 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423332 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423335 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423337 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:26:47.425811 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423340 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423343 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423345 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423348 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423350 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423353 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423355 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423358 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423360 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423363 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423366 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423369 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423372 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423374 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423378 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423381 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423383 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423386 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423388 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423391 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:26:47.426372 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423394 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423396 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423399 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423401 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423404 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423407 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423409 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423411 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423414 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423417 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423419 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423422 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423424 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423911 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423916 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423920 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423923 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423925 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423928 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423931 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:26:47.427138 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423934 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423937 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423940 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423943 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423946 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423949 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423951 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423954 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423957 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423960 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423962 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423965 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423968 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423970 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423973 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423976 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423979 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423982 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423985 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423987 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:26:47.427638 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423990 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423993 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423995 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.423998 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424001 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424003 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424006 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424008 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424012 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424014 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424017 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424019 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424022 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424024 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424027 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424029 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424032 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424034 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424037 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424039 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:26:47.428124 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424042 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424044 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424046 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424049 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424052 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424054 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424057 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424061 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424064 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424068 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424071 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424074 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424077 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424079 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424082 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424084 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424087 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424090 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424092 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:26:47.428644 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424095 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424097 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424100 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424102 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424105 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424108 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424110 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424113 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424116 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424118 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424121 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424123 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424125 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424128 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424130 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424133 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424136 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424138 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424141 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:26:47.429085 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424143 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424146 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424151 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424153 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424156 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424159 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424161 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424164 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424168 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424171 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424174 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424177 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424179 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424182 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424185 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424188 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424190 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424192 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424195 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424197 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:26:47.429580 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.424200 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425450 2566 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425462 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425469 2566 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425475 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425480 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425483 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425488 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425492 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425496 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425499 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425503 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425506 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425510 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425514 2566 flags.go:64] FLAG: --cgroup-root="" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425517 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425520 2566 flags.go:64] FLAG: --client-ca-file="" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425524 2566 flags.go:64] FLAG: --cloud-provider="external" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425527 2566 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425531 2566 flags.go:64] FLAG: --cluster-domain="" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425534 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425538 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425541 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425545 2566 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 17:26:47.430154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425550 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425553 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425557 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425560 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425563 2566 flags.go:64] FLAG: --contention-profiling="false" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425567 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425570 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425573 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425576 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425580 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425584 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425587 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425590 2566 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425593 2566 flags.go:64] FLAG: --enable-server="true" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425596 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425601 2566 flags.go:64] FLAG: --event-burst="100" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425605 2566 flags.go:64] FLAG: --event-qps="50" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425608 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425611 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425614 2566 flags.go:64] FLAG: --eviction-hard="" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425619 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425622 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425625 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425628 2566 flags.go:64] FLAG: --eviction-soft="" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425631 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 17:26:47.430706 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425635 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425638 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425642 2566 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425645 2566 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425649 2566 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425652 2566 flags.go:64] FLAG: --feature-gates="" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425656 2566 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425659 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425663 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425666 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425669 2566 flags.go:64] FLAG: --healthz-port="10248" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425673 2566 flags.go:64] FLAG: --help="false" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425675 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425679 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425682 2566 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425685 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425688 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425692 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425695 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425698 2566 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425701 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425704 2566 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425707 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425710 2566 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 17:26:47.431328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425713 2566 flags.go:64] FLAG: --kube-reserved="" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425717 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425719 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425722 2566 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425725 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425728 2566 flags.go:64] FLAG: --lock-file="" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425731 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425734 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425737 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425743 2566 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425749 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425752 2566 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425755 2566 flags.go:64] FLAG: --logging-format="text" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425759 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425762 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425765 2566 flags.go:64] FLAG: --manifest-url="" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425768 2566 flags.go:64] FLAG: --manifest-url-header="" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425773 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425776 2566 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425779 2566 flags.go:64] FLAG: --max-pods="110" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425783 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425786 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425789 2566 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425793 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425796 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 17:26:47.431894 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425799 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425802 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425810 2566 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425813 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425816 2566 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425820 2566 flags.go:64] FLAG: --pod-cidr="" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425823 2566 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425826 2566 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425830 2566 flags.go:64] FLAG: --pods-per-core="0" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425832 2566 flags.go:64] FLAG: --port="10250" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425835 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425838 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c5258cd750d2d2c5" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425842 2566 flags.go:64] FLAG: --qos-reserved="" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425845 2566 flags.go:64] FLAG: --read-only-port="10255" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425848 2566 flags.go:64] FLAG: --register-node="true" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425851 2566 flags.go:64] FLAG: --register-with-taints="" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425855 2566 flags.go:64] FLAG: --registry-burst="10" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425862 2566 flags.go:64] FLAG: --registry-qps="5" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425866 2566 flags.go:64] FLAG: --reserved-cpus="" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425869 2566 flags.go:64] FLAG: --reserved-memory="" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425873 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425876 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425879 2566 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425882 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425885 2566 flags.go:64] FLAG: --runonce="false" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425888 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 17:26:47.432506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425892 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425895 2566 flags.go:64] FLAG: --seccomp-default="false" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425898 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425901 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425904 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425907 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425911 2566 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425914 2566 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425917 2566 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425921 2566 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425924 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425927 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425931 2566 flags.go:64] FLAG: --system-cgroups="" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425934 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425939 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425942 2566 flags.go:64] FLAG: --tls-cert-file="" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425945 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425950 2566 flags.go:64] FLAG: --tls-min-version="" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425953 2566 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425956 2566 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425959 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425962 2566 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425965 2566 flags.go:64] FLAG: --v="2" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425971 2566 flags.go:64] FLAG: --version="false" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425975 2566 flags.go:64] FLAG: --vmodule="" Oct 31 17:26:47.433099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425980 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.425984 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427393 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427406 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427410 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427414 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427417 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427420 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427423 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427429 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427431 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427434 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427437 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427440 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427443 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427446 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427451 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427454 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427457 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427460 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:26:47.433801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427462 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427465 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427470 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427473 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427476 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427479 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427482 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427484 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427487 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427490 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427493 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427496 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427499 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427507 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427513 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427516 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427519 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427522 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427524 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:26:47.434351 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427527 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427530 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427533 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427535 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427538 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427540 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427543 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427546 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427551 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427559 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427561 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427564 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427567 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427569 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427572 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427574 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427577 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427580 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427582 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:26:47.434792 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427585 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427588 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427593 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427595 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427599 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427601 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427604 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427607 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427615 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427618 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427623 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427627 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427631 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427634 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427639 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427642 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427645 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427648 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427651 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:26:47.435238 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427654 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427657 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427659 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427664 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427666 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427669 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427672 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427674 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427680 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427682 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427685 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427688 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427690 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427693 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427696 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427698 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427701 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427704 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427707 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427709 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:26:47.435735 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427712 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427718 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427726 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427729 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427732 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427737 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427740 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427743 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427746 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427748 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427751 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.427791 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427868 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427873 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427877 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427881 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:26:47.436230 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427884 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427887 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427890 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427892 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427895 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427898 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427900 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427906 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427909 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427912 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427915 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427918 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427922 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427926 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427929 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427931 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427934 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427937 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427939 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427950 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:26:47.436634 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427954 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427956 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427959 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427962 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427964 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427967 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427969 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427972 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427974 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427977 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427980 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427984 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427989 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427991 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427994 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427996 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.427999 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428002 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428006 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:26:47.437086 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428009 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428011 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428015 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428017 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428020 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428023 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428028 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428031 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428033 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428036 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428039 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428041 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428044 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428058 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428062 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428064 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428067 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428070 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428075 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428077 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:26:47.437534 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428080 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428082 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428093 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428096 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428100 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428102 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428105 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428107 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428110 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428113 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428115 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428122 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428125 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428128 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428131 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428133 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428136 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428139 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428143 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:26:47.438003 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428145 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428148 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428151 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428153 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428158 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428162 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428166 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428175 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428178 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428181 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428184 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428187 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428189 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428192 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428195 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428198 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428200 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428207 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428209 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:26:47.438461 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428212 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428215 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428217 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428220 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.428223 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.435165 2566 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.435184 2566 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.435201 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.435207 2566 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435281 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435288 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435291 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435294 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435297 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435299 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435302 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435305 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435308 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435310 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:26:47.438926 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435313 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435316 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435319 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435321 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435325 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435329 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435332 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435335 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435338 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435342 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435345 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435348 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435351 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435354 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435357 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435359 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435362 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435365 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435367 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:26:47.439384 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435371 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435374 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435377 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435379 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435382 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435385 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435387 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435390 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435392 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435395 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435398 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435401 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435403 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435406 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435409 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435411 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435414 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435417 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435419 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435422 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:26:47.439832 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435425 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435427 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435430 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435433 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435435 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435438 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435441 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435443 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435445 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435448 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435451 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435453 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435456 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435459 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435461 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435464 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435466 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435469 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435471 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435474 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:26:47.440330 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435477 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435480 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435483 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435486 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435489 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435491 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435494 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435496 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435499 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435501 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435504 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435506 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435509 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435511 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435514 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435516 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435519 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435522 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435524 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435527 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:26:47.440801 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435529 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435532 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435535 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435538 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435541 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435544 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435547 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435550 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435552 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435555 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435557 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435561 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435565 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435568 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435570 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435573 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:26:47.441311 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435576 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.435605 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435744 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435748 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435751 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435755 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435758 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435760 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435764 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435766 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435769 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435772 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435774 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435777 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435779 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435782 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:26:47.441683 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435785 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435787 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435790 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435793 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435796 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435798 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435801 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435803 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435806 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435809 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435812 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435815 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435818 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435820 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435823 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435826 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435828 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435831 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435833 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435836 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:26:47.443130 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435838 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435841 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435844 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435846 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435849 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435852 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435855 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435857 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435860 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435862 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435865 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435867 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435870 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435872 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435875 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435877 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435880 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435883 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435885 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:26:47.443942 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435888 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435890 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435893 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435896 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435899 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435901 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435904 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435906 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435909 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435911 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435914 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435916 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435918 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435921 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435924 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435927 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435929 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435931 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435934 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435937 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:26:47.444472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435939 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435942 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435944 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435947 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435949 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435951 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435954 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435956 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435959 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435961 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435965 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435969 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435973 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435976 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435979 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435982 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435985 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435988 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:26:47.444968 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435991 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435993 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435996 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.435999 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436002 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436004 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436007 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436009 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436012 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436015 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436017 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436020 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436022 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436025 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436028 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.436050 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 17:26:47.445391 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436082 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436087 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436090 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436093 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436096 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436098 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436101 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436104 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436106 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436108 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436111 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436114 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436117 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436119 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436122 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436125 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436128 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436131 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436134 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:26:47.445912 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436136 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436138 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436141 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436144 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436146 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436149 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436152 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436155 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436157 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436160 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436162 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436165 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436167 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436170 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436172 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436175 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436178 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436180 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436183 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436185 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:26:47.446383 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436188 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436190 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436193 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436195 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436198 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436203 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436205 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436208 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436211 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436214 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436216 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436218 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436221 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436223 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436226 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436228 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436231 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436233 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436236 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:26:47.446904 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436239 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436242 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436246 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436263 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436268 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436271 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436274 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436277 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436279 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436282 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436285 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436287 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436290 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436292 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436295 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436298 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436300 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436303 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436308 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:26:47.447472 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436310 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436313 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436315 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436319 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436322 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436325 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436327 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436330 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436333 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436335 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436338 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436340 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436343 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436345 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436348 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436350 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436352 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436355 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436357 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436360 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:26:47.447924 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436362 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436365 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436367 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436371 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436373 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436376 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436378 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436381 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:47.436383 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.436452 2566 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.438640 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.439613 2566 server.go:1014] "Starting client certificate rotation" Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.439711 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.439781 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.442140 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 17:26:47.448615 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.443942 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 17:26:47.458019 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.458000 2566 log.go:25] "Validated CRI v1 runtime API" Oct 31 17:26:47.461961 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.461942 2566 log.go:25] "Validated CRI v1 image API" Oct 31 17:26:47.463948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.463923 2566 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 17:26:47.470727 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.470702 2566 fs.go:135] Filesystem UUIDs: map[37ae1589-8032-4418-aa19-6a80e4e1c11e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b23beb05-70ea-41c7-b37b-98759e30376f:/dev/nvme0n1p4] Oct 31 17:26:47.470816 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.470724 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 17:26:47.476377 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.476235 2566 manager.go:217] Machine: {Timestamp:2026-10-31 17:26:47.474725606 +0000 UTC m=+0.371557204 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3030314 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2193e5e020c7c03bc4b733ed2f13b4 SystemUUID:ec2193e5-e020-c7c0-3bc4-b733ed2f13b4 BootID:33afd093-d3e1-4d3b-ba37-70e9e8cc1013 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:f9:ad:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:f9:ad:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:da:d5:44:94:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 17:26:47.477035 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.477022 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 17:26:47.477190 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.477173 2566 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 17:26:47.479247 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.479216 2566 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 17:26:47.479447 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.479269 2566 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-107.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 17:26:47.479490 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.479451 2566 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 17:26:47.479490 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.479462 2566 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 17:26:47.479551 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.479514 2566 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 17:26:47.479551 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.479531 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 17:26:47.480523 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.480509 2566 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 17:26:47.484534 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.484519 2566 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 17:26:47.484681 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.484671 2566 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 17:26:47.485557 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.485546 2566 kubelet.go:493] "Attempting to sync node with API server" Oct 31 17:26:47.485591 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.485563 2566 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 17:26:47.485591 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.485578 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 17:26:47.485704 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.485591 2566 kubelet.go:405] "Adding apiserver pod source" Oct 31 17:26:47.485704 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.485601 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 17:26:47.486026 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.486003 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 17:26:47.488377 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.488353 2566 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 17:26:47.494201 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.494189 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 17:26:47.495146 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.495134 2566 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 17:26:47.495196 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.495165 2566 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 17:26:47.495392 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.495377 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 17:26:47.495429 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.495403 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 17:26:47.495429 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.495416 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 17:26:47.495480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.495428 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 17:26:47.495480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.495440 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 17:26:47.495480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.495454 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 17:26:47.495998 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.495988 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 17:26:47.496028 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.495999 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 17:26:47.496028 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.496006 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 17:26:47.496028 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.496012 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 17:26:47.496028 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.496021 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 17:26:47.496217 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.496030 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 17:26:47.496635 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.496624 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 17:26:47.496672 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.496637 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 17:26:47.500878 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.500862 2566 server.go:1263] "Started kubelet" Oct 31 17:26:47.501004 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.500955 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 17:26:47.501067 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.501024 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 17:26:47.501067 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.501038 2566 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 17:26:47.501608 ip-10-0-139-107 systemd[1]: Started Kubernetes Kubelet. Oct 31 17:26:47.502499 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.502482 2566 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 17:26:47.503341 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.503326 2566 server.go:317] "Adding debug handlers to kubelet server" Oct 31 17:26:47.507018 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:47.506998 2566 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 17:26:47.508461 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.508427 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 17:26:47.509742 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.509724 2566 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 17:26:47.510349 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:47.510017 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Oct 31 17:26:47.510498 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.510482 2566 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 17:26:47.510575 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.510501 2566 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 17:26:47.510610 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.510592 2566 reconstruct.go:97] "Volume reconstruction finished" Oct 31 17:26:47.510610 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.510600 2566 reconciler.go:29] "Reconciler: start to sync state" Oct 31 17:26:47.510683 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.510621 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 17:26:47.512331 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.512312 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 17:26:47.512331 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.512330 2566 factory.go:55] Registering systemd factory Oct 31 17:26:47.512469 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.512339 2566 factory.go:223] Registration of the systemd container factory successfully Oct 31 17:26:47.512585 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.512571 2566 factory.go:153] Registering CRI-O factory Oct 31 17:26:47.512619 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.512590 2566 factory.go:223] Registration of the crio container factory successfully Oct 31 17:26:47.512619 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.512607 2566 factory.go:103] Registering Raw factory Oct 31 17:26:47.512674 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.512620 2566 manager.go:1199] Started watching for new ooms in manager Oct 31 17:26:47.513615 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.513599 2566 manager.go:319] Starting recovery of all containers Oct 31 17:26:47.521033 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.520989 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 17:26:47.527831 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.527814 2566 manager.go:324] Recovery completed Oct 31 17:26:47.529265 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:47.529229 2566 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 17:26:47.533196 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.533180 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 17:26:47.535028 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.535007 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientMemory" Oct 31 17:26:47.535107 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.535045 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 17:26:47.535107 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.535059 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientPID" Oct 31 17:26:47.535549 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.535529 2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 17:26:47.535626 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.535594 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bjdj4" Oct 31 17:26:47.535724 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.535711 2566 cpu_manager.go:226] "Starting" policy="none" Oct 31 17:26:47.535777 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.535723 2566 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 17:26:47.535777 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.535744 2566 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 17:26:47.535777 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.535745 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 17:26:47.536583 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.536559 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientMemory" Oct 31 17:26:47.536651 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.536606 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 17:26:47.536651 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.536618 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientPID" Oct 31 17:26:47.537988 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.537974 2566 policy_none.go:50] "Start" Oct 31 17:26:47.538029 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.537996 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 17:26:47.538029 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.538011 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 17:26:47.539374 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.539362 2566 policy_none.go:44] "Start" Oct 31 17:26:47.545433 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.545401 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-107.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 17:26:47.546503 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:47.545460 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-107.ec2.internal.18e3ad496d41c70c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-107.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-107.ec2.internal,},FirstTimestamp:2026-10-31 17:26:47.500834572 +0000 UTC m=+0.397666171,LastTimestamp:2026-10-31 17:26:47.500834572 +0000 UTC m=+0.397666171,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-107.ec2.internal,}" Oct 31 17:26:47.561210 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.561190 2566 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 17:26:47.562823 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.562803 2566 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 17:26:47.563922 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.563888 2566 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 17:26:47.566200 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.566188 2566 manager.go:341] "Starting Device Plugin manager" Oct 31 17:26:47.566279 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:47.566266 2566 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 17:26:47.566336 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.566280 2566 server.go:87] "Starting device plugin registration server" Oct 31 17:26:47.566554 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.566538 2566 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 17:26:47.566644 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.566555 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 17:26:47.566644 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.566633 2566 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 17:26:47.566730 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.566710 2566 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 17:26:47.566730 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.566718 2566 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 17:26:47.567172 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:47.567150 2566 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 17:26:47.567247 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:47.567194 2566 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-107.ec2.internal\" not found" Oct 31 17:26:47.592652 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.592617 2566 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 17:26:47.642047 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.642010 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 17:26:47.642047 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.642041 2566 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 17:26:47.643369 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.642066 2566 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 17:26:47.643369 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:47.642101 2566 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 17:26:47.658697 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.658674 2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 17:26:47.666991 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.666923 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 17:26:47.667857 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.667838 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientMemory" Oct 31 17:26:47.667947 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.667870 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 17:26:47.667947 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.667886 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientPID" Oct 31 17:26:47.667947 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.667920 2566 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.687421 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.687398 2566 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.687486 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:47.687428 2566 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-107.ec2.internal\": node \"ip-10-0-139-107.ec2.internal\" not found" Oct 31 17:26:47.743020 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.742990 2566 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal"] Oct 31 17:26:47.744680 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.744663 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.745621 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.745606 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.785479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.785324 2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.788804 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.788788 2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.810066 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.810041 2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 17:26:47.810207 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.810112 2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 17:26:47.812300 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.812276 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8765d5fdccb7ed0c5cd03e206e2ad979-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"8765d5fdccb7ed0c5cd03e206e2ad979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.812357 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.812309 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8765d5fdccb7ed0c5cd03e206e2ad979-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"8765d5fdccb7ed0c5cd03e206e2ad979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.812357 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.812326 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8389bef8c4009e269aafe36de827cd22-config\") pod \"kube-apiserver-proxy-ip-10-0-139-107.ec2.internal\" (UID: \"8389bef8c4009e269aafe36de827cd22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.912565 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.912531 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8765d5fdccb7ed0c5cd03e206e2ad979-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"8765d5fdccb7ed0c5cd03e206e2ad979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.912565 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.912561 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8765d5fdccb7ed0c5cd03e206e2ad979-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"8765d5fdccb7ed0c5cd03e206e2ad979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.912791 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.912581 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8389bef8c4009e269aafe36de827cd22-config\") pod \"kube-apiserver-proxy-ip-10-0-139-107.ec2.internal\" (UID: \"8389bef8c4009e269aafe36de827cd22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.912791 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.912636 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8389bef8c4009e269aafe36de827cd22-config\") pod \"kube-apiserver-proxy-ip-10-0-139-107.ec2.internal\" (UID: \"8389bef8c4009e269aafe36de827cd22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.912791 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.912646 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8765d5fdccb7ed0c5cd03e206e2ad979-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"8765d5fdccb7ed0c5cd03e206e2ad979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:26:47.912791 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:47.912647 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8765d5fdccb7ed0c5cd03e206e2ad979-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"8765d5fdccb7ed0c5cd03e206e2ad979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:26:48.115865 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.115822 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:26:48.116071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.115866 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:26:48.439000 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.438909 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 17:26:48.439545 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.439047 2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 17:26:48.439545 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.439046 2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 17:26:48.439545 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.439088 2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 17:26:48.486099 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.486063 2566 apiserver.go:52] "Watching apiserver" Oct 31 17:26:48.509583 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.509550 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 17:26:48.537076 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.537044 2566 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 17:26:48.537451 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.537427 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4","openshift-image-registry/node-ca-9789n","openshift-multus/network-metrics-daemon-9zfqw","kube-system/global-pull-secret-syncer-swbqv","kube-system/konnectivity-agent-v9fs8","openshift-cluster-node-tuning-operator/tuned-cpz9p","openshift-dns/node-resolver-58mtk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal","openshift-multus/multus-862p4","openshift-multus/multus-additional-cni-plugins-2klfh","openshift-network-operator/iptables-alerter-cssnv","openshift-ovn-kubernetes/ovnkube-node-tdrsm","openshift-network-diagnostics/network-check-target-cd9s8","kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal"] Oct 31 17:26:48.538980 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.538956 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.539232 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.539203 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 17:21:48 +0000 UTC" deadline="2028-07-24 09:06:46.352856458 +0000 UTC" Oct 31 17:26:48.539308 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.539233 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15159h39m57.813626424s" Oct 31 17:26:48.540539 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.540515 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9789n" Oct 31 17:26:48.541863 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.541842 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:48.541970 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:48.541916 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:26:48.543238 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.543220 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:48.543349 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:48.543299 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:26:48.544727 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.544713 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:26:48.546136 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.546116 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.547661 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.547644 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-58mtk" Oct 31 17:26:48.549106 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.549088 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-862p4" Oct 31 17:26:48.550576 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.550562 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.552419 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.552401 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:26:48.555039 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.555019 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.556635 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.556612 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 17:26:48.556635 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.556632 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9gvm6\"" Oct 31 17:26:48.556791 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.556678 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 17:26:48.560217 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.557116 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:26:48.560217 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.557192 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 17:26:48.560217 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:48.557223 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:26:48.560217 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.557559 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 17:26:48.560217 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.557681 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 17:26:48.560217 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.557909 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zv4z8\"" Oct 31 17:26:48.560217 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.558016 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 17:26:48.569954 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.569934 2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 17:26:48.570030 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.570003 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f7p7f" Oct 31 17:26:48.611928 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.611900 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 17:26:48.617825 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.617797 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txcnm\" (UniqueName: \"kubernetes.io/projected/64a53072-d4ff-45d1-ab01-249f63874e30-kube-api-access-txcnm\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.617974 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.617838 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovn-node-metrics-cert\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.617974 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.617870 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq4bl\" (UniqueName: \"kubernetes.io/projected/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-kube-api-access-qq4bl\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.617974 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.617904 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76940cd0-ef63-4e10-965e-e871ab476281-konnectivity-ca\") pod \"konnectivity-agent-v9fs8\" (UID: \"76940cd0-ef63-4e10-965e-e871ab476281\") " pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:26:48.617974 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.617929 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-system-cni-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.617974 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.617946 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4x98\" (UniqueName: \"kubernetes.io/projected/d30a63cf-3116-466f-8e4d-c77f6eaacb37-kube-api-access-n4x98\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.617979 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-kubernetes\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618004 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-cni-bin\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618019 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-modprobe-d\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618034 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-run\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618056 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618081 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrmhn\" (UniqueName: \"kubernetes.io/projected/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-kube-api-access-nrmhn\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618112 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618141 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovnkube-config\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618162 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-socket-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618179 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bldpx\" (UniqueName: \"kubernetes.io/projected/016409a4-3533-46e9-970d-4154787d4924-kube-api-access-bldpx\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618212 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/159b2703-3b66-4ddc-8762-b4546b942282-hosts-file\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618230 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-netns\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.618246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618247 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-tmp\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618269 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-env-overrides\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618313 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618339 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-sys-fs\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618355 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-kubelet\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618379 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/806f6a75-275f-48d2-ae95-03e1a5e46515-host\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618392 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-sys\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618414 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-cni-netd\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618436 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-k8s-cni-cncf-io\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618497 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-hostroot\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618514 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-conf-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618553 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysctl-d\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618569 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618587 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g46kf\" (UniqueName: \"kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf\") pod \"network-check-target-cd9s8\" (UID: \"bcd594b6-86d2-47ff-bc81-03bd655d3954\") " pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618604 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64a53072-d4ff-45d1-ab01-249f63874e30-multus-daemon-config\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618622 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-lib-modules\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.618921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618636 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-kubelet\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618651 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovnkube-script-lib\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618673 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdrvg\" (UniqueName: \"kubernetes.io/projected/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-kube-api-access-pdrvg\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618694 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-host\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618708 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-ovn\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618723 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-cni-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618740 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76940cd0-ef63-4e10-965e-e871ab476281-agent-certs\") pod \"konnectivity-agent-v9fs8\" (UID: \"76940cd0-ef63-4e10-965e-e871ab476281\") " pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618800 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618819 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618835 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwxk9\" (UniqueName: \"kubernetes.io/projected/806f6a75-275f-48d2-ae95-03e1a5e46515-kube-api-access-wwxk9\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618854 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-systemd\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618870 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-tuned\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618915 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-systemd-units\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618944 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618970 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159b2703-3b66-4ddc-8762-b4546b942282-tmp-dir\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.618994 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64a53072-d4ff-45d1-ab01-249f63874e30-cni-binary-copy\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.619480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619021 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619053 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-host-slash\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619080 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-var-lib-kubelet\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619107 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-node-log\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619130 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bec5b1da-4675-4713-9e51-9e7be00b5ba0-dbus\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619153 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8d9h\" (UniqueName: \"kubernetes.io/projected/159b2703-3b66-4ddc-8762-b4546b942282-kube-api-access-c8d9h\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619178 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cnibin\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-systemd\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619227 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-var-lib-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619250 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-run-ovn-kubernetes\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619275 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-registration-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-socket-dir-parent\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619323 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-cni-multus\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619369 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-etc-kubernetes\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619388 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cni-binary-copy\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619404 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-iptables-alerter-script\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:26:48.619948 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619421 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysconfig\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619437 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysctl-conf\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619455 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-cnibin\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619579 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blr7j\" (UniqueName: \"kubernetes.io/projected/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-kube-api-access-blr7j\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619598 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-slash\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619622 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-log-socket\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619639 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bec5b1da-4675-4713-9e51-9e7be00b5ba0-kubelet-config\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619656 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-system-cni-dir\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619681 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/806f6a75-275f-48d2-ae95-03e1a5e46515-serviceca\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619698 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-run-netns\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619714 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-etc-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619729 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619745 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-device-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619761 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-cni-bin\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.620435 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.619781 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-multus-certs\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.666325 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.666291 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 17:26:48.666741 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.666719 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 17:26:48.666857 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.666741 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 17:26:48.666857 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.666778 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 17:26:48.666857 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.666777 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 17:26:48.667013 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.666875 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 17:26:48.667013 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.666876 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 17:26:48.667013 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.666926 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 17:26:48.667013 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.666984 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 17:26:48.667344 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.667296 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 17:26:48.667344 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.667313 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 17:26:48.667344 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.667341 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5bhk8\"" Oct 31 17:26:48.667493 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.667331 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k6pvq\"" Oct 31 17:26:48.667493 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.667352 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 17:26:48.667493 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.667418 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 17:26:48.667493 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.667422 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 17:26:48.667493 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.667454 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 17:26:48.681818 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.681796 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 17:26:48.681818 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.681812 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 17:26:48.681963 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.681855 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 17:26:48.682289 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.682275 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 17:26:48.682378 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.682323 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xbzzg\"" Oct 31 17:26:48.682378 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.682349 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vvnlg\"" Oct 31 17:26:48.682480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.682378 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g2t4b\"" Oct 31 17:26:48.682480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.682396 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 17:26:48.682480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.682406 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 17:26:48.682480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.682426 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8d7mp\"" Oct 31 17:26:48.682480 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.682432 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x4c5q\"" Oct 31 17:26:48.697754 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:48.697729 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8389bef8c4009e269aafe36de827cd22.slice/crio-d5989dbdb2d1c5d28a2b391e91e571a62d76150c1afe36bf4435b89aedd32b6c WatchSource:0}: Error finding container d5989dbdb2d1c5d28a2b391e91e571a62d76150c1afe36bf4435b89aedd32b6c: Status 404 returned error can't find the container with id d5989dbdb2d1c5d28a2b391e91e571a62d76150c1afe36bf4435b89aedd32b6c Oct 31 17:26:48.699128 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:48.699108 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8765d5fdccb7ed0c5cd03e206e2ad979.slice/crio-49daaf0fda866993155a9e9a1e9d051c758c64ba9cde917ea68014e7b17ca97a WatchSource:0}: Error finding container 49daaf0fda866993155a9e9a1e9d051c758c64ba9cde917ea68014e7b17ca97a: Status 404 returned error can't find the container with id 49daaf0fda866993155a9e9a1e9d051c758c64ba9cde917ea68014e7b17ca97a Oct 31 17:26:48.705461 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.705442 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 17:26:48.721053 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721027 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-system-cni-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.721200 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721060 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4x98\" (UniqueName: \"kubernetes.io/projected/d30a63cf-3116-466f-8e4d-c77f6eaacb37-kube-api-access-n4x98\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:48.721200 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721080 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-kubernetes\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.721200 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721100 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-cni-bin\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.721200 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721123 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-modprobe-d\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.721200 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721183 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-system-cni-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721243 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-modprobe-d\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721245 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-kubernetes\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721278 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-run\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721308 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721333 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrmhn\" (UniqueName: \"kubernetes.io/projected/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-kube-api-access-nrmhn\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721356 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721374 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovnkube-config\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721377 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-run\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721398 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-socket-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721424 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldpx\" (UniqueName: \"kubernetes.io/projected/016409a4-3533-46e9-970d-4154787d4924-kube-api-access-bldpx\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721448 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/159b2703-3b66-4ddc-8762-b4546b942282-hosts-file\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721473 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-netns\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.721479 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:48.721481 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-tmp\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721520 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-env-overrides\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:48.721546 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs podName:d30a63cf-3116-466f-8e4d-c77f6eaacb37 nodeName:}" failed. No retries permitted until 2026-10-31 17:26:49.221524378 +0000 UTC m=+2.118355985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs") pod "network-metrics-daemon-9zfqw" (UID: "d30a63cf-3116-466f-8e4d-c77f6eaacb37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721578 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721606 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-sys-fs\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721631 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-kubelet\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:48.721655 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721178 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-cni-bin\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721678 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/806f6a75-275f-48d2-ae95-03e1a5e46515-host\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721725 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-kubelet\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721654 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/806f6a75-275f-48d2-ae95-03e1a5e46515-host\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721747 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/159b2703-3b66-4ddc-8762-b4546b942282-hosts-file\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:48.721761 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret podName:bec5b1da-4675-4713-9e51-9e7be00b5ba0 nodeName:}" failed. No retries permitted until 2026-10-31 17:26:49.22174177 +0000 UTC m=+2.118573375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret") pod "global-pull-secret-syncer-swbqv" (UID: "bec5b1da-4675-4713-9e51-9e7be00b5ba0") : object "kube-system"/"original-pull-secret" not registered Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721777 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-netns\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721789 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-sys-fs\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721805 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-sys\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.722105 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721832 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-cni-netd\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721860 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-k8s-cni-cncf-io\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721886 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-hostroot\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721908 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-conf-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721931 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysctl-d\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721961 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721964 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-env-overrides\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.721908 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-socket-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722006 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-k8s-cni-cncf-io\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722007 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-conf-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722029 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46kf\" (UniqueName: \"kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf\") pod \"network-check-target-cd9s8\" (UID: \"bcd594b6-86d2-47ff-bc81-03bd655d3954\") " pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722043 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722068 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64a53072-d4ff-45d1-ab01-249f63874e30-multus-daemon-config\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722076 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovnkube-config\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722087 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-cni-netd\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722009 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-sys\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722097 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-lib-modules\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-kubelet\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.723038 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722134 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysctl-d\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722146 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722085 2566 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722071 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-hostroot\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovnkube-script-lib\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722194 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-kubelet\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722205 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdrvg\" (UniqueName: \"kubernetes.io/projected/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-kube-api-access-pdrvg\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722229 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-lib-modules\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722265 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-host\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722293 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-ovn\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722318 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-cni-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722366 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76940cd0-ef63-4e10-965e-e871ab476281-agent-certs\") pod \"konnectivity-agent-v9fs8\" (UID: \"76940cd0-ef63-4e10-965e-e871ab476281\") " pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722383 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-ovn\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722399 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722429 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722429 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-cni-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722446 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-host\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722476 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwxk9\" (UniqueName: \"kubernetes.io/projected/806f6a75-275f-48d2-ae95-03e1a5e46515-kube-api-access-wwxk9\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:26:48.723962 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722511 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-systemd\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722534 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-tuned\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722575 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-systemd-units\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722599 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722668 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159b2703-3b66-4ddc-8762-b4546b942282-tmp-dir\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722702 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64a53072-d4ff-45d1-ab01-249f63874e30-cni-binary-copy\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722732 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovnkube-script-lib\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722753 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-host-slash\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722836 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-host-slash\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722842 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722875 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-var-lib-kubelet\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722901 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-node-log\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722924 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bec5b1da-4675-4713-9e51-9e7be00b5ba0-dbus\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722949 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8d9h\" (UniqueName: \"kubernetes.io/projected/159b2703-3b66-4ddc-8762-b4546b942282-kube-api-access-c8d9h\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722973 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cnibin\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722997 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-systemd\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.724763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723022 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-var-lib-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723045 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-run-ovn-kubernetes\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723063 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-registration-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723100 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-socket-dir-parent\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723124 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159b2703-3b66-4ddc-8762-b4546b942282-tmp-dir\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723126 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-cni-multus\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723166 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-etc-kubernetes\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723199 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cni-binary-copy\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723222 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-iptables-alerter-script\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723245 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysconfig\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723267 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysctl-conf\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723289 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-cnibin\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723310 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blr7j\" (UniqueName: \"kubernetes.io/projected/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-kube-api-access-blr7j\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723361 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-slash\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723368 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-systemd\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723383 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-log-socket\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.725634 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723413 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bec5b1da-4675-4713-9e51-9e7be00b5ba0-kubelet-config\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723450 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-etc-kubernetes\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723478 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bec5b1da-4675-4713-9e51-9e7be00b5ba0-kubelet-config\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723483 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-system-cni-dir\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723522 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/806f6a75-275f-48d2-ae95-03e1a5e46515-serviceca\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723549 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-run-netns\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723552 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysconfig\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723576 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-etc-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723603 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723630 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-device-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-cni-bin\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723674 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysctl-conf\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723681 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-multus-certs\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723708 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-multus-certs\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723712 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txcnm\" (UniqueName: \"kubernetes.io/projected/64a53072-d4ff-45d1-ab01-249f63874e30-kube-api-access-txcnm\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723717 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-cnibin\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723749 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovn-node-metrics-cert\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723764 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-slash\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.726527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723777 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq4bl\" (UniqueName: \"kubernetes.io/projected/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-kube-api-access-qq4bl\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723769 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-log-socket\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.722667 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64a53072-d4ff-45d1-ab01-249f63874e30-multus-daemon-config\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723168 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-cni-multus\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723841 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76940cd0-ef63-4e10-965e-e871ab476281-konnectivity-ca\") pod \"konnectivity-agent-v9fs8\" (UID: \"76940cd0-ef63-4e10-965e-e871ab476281\") " pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723857 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cnibin\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723897 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64a53072-d4ff-45d1-ab01-249f63874e30-cni-binary-copy\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723919 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-etc-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723918 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723953 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-cni-bin\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723967 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-device-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.723983 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bec5b1da-4675-4713-9e51-9e7be00b5ba0-dbus\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.724096 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.724130 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-system-cni-dir\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.724143 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-run-netns\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.724664 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-var-lib-kubelet\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.724733 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-systemd-units\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.724905 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/806f6a75-275f-48d2-ae95-03e1a5e46515-serviceca\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:26:48.727446 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.725032 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-node-log\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.725417 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-registration-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.725430 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-socket-dir-parent\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.725470 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-run-ovn-kubernetes\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.725479 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-systemd\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.725530 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-var-lib-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.726147 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cni-binary-copy\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.726388 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76940cd0-ef63-4e10-965e-e871ab476281-konnectivity-ca\") pod \"konnectivity-agent-v9fs8\" (UID: \"76940cd0-ef63-4e10-965e-e871ab476281\") " pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.726575 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-tuned\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.726584 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.726828 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-iptables-alerter-script\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.726996 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-tmp\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.727198 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76940cd0-ef63-4e10-965e-e871ab476281-agent-certs\") pod \"konnectivity-agent-v9fs8\" (UID: \"76940cd0-ef63-4e10-965e-e871ab476281\") " pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:26:48.728246 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.727497 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovn-node-metrics-cert\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.741677 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:48.741648 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 17:26:48.741677 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:48.741673 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 17:26:48.741847 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:48.741685 2566 projected.go:196] Error preparing data for projected volume kube-api-access-g46kf for pod openshift-network-diagnostics/network-check-target-cd9s8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:26:48.741847 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:48.741739 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf podName:bcd594b6-86d2-47ff-bc81-03bd655d3954 nodeName:}" failed. No retries permitted until 2026-10-31 17:26:49.241722856 +0000 UTC m=+2.138554463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g46kf" (UniqueName: "kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf") pod "network-check-target-cd9s8" (UID: "bcd594b6-86d2-47ff-bc81-03bd655d3954") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:26:48.743666 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.743633 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4x98\" (UniqueName: \"kubernetes.io/projected/d30a63cf-3116-466f-8e4d-c77f6eaacb37-kube-api-access-n4x98\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:48.743781 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.743686 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrmhn\" (UniqueName: \"kubernetes.io/projected/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-kube-api-access-nrmhn\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:26:48.744992 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.744955 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdrvg\" (UniqueName: \"kubernetes.io/projected/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-kube-api-access-pdrvg\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.745113 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.744985 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldpx\" (UniqueName: \"kubernetes.io/projected/016409a4-3533-46e9-970d-4154787d4924-kube-api-access-bldpx\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.745207 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.745186 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blr7j\" (UniqueName: \"kubernetes.io/projected/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-kube-api-access-blr7j\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.745490 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.745469 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwxk9\" (UniqueName: \"kubernetes.io/projected/806f6a75-275f-48d2-ae95-03e1a5e46515-kube-api-access-wwxk9\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:26:48.745851 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.745827 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8d9h\" (UniqueName: \"kubernetes.io/projected/159b2703-3b66-4ddc-8762-b4546b942282-kube-api-access-c8d9h\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:26:48.745851 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.745846 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txcnm\" (UniqueName: \"kubernetes.io/projected/64a53072-d4ff-45d1-ab01-249f63874e30-kube-api-access-txcnm\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:26:48.747067 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.747047 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq4bl\" (UniqueName: \"kubernetes.io/projected/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-kube-api-access-qq4bl\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:48.875348 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.875291 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:26:48.886476 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:48.886448 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016409a4_3533_46e9_970d_4154787d4924.slice/crio-fc3b96f14c433569b9d1c4ad0849f0e7828dba0d2d8806b7fcb408977713ce73 WatchSource:0}: Error finding container fc3b96f14c433569b9d1c4ad0849f0e7828dba0d2d8806b7fcb408977713ce73: Status 404 returned error can't find the container with id fc3b96f14c433569b9d1c4ad0849f0e7828dba0d2d8806b7fcb408977713ce73 Oct 31 17:26:48.887136 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.887114 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9789n" Oct 31 17:26:48.895650 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.895617 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:26:48.895807 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:48.895781 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806f6a75_275f_48d2_ae95_03e1a5e46515.slice/crio-f2a476a670e5c7ff9a8ec1b564638865478650ecf8fd0f9596082da8b9bf5f69 WatchSource:0}: Error finding container f2a476a670e5c7ff9a8ec1b564638865478650ecf8fd0f9596082da8b9bf5f69: Status 404 returned error can't find the container with id f2a476a670e5c7ff9a8ec1b564638865478650ecf8fd0f9596082da8b9bf5f69 Oct 31 17:26:48.904547 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:48.904519 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76940cd0_ef63_4e10_965e_e871ab476281.slice/crio-94a1f08db33c8a171e666896062ed0eae9d3742fd533efb76c3c74ca5cc71816 WatchSource:0}: Error finding container 94a1f08db33c8a171e666896062ed0eae9d3742fd533efb76c3c74ca5cc71816: Status 404 returned error can't find the container with id 94a1f08db33c8a171e666896062ed0eae9d3742fd533efb76c3c74ca5cc71816 Oct 31 17:26:48.907946 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.907927 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:26:48.920982 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.920944 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-58mtk" Oct 31 17:26:48.929697 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:48.929671 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159b2703_3b66_4ddc_8762_b4546b942282.slice/crio-3f0f6f84b766e8e9c27ae5a1b15af9c9c97e80544f343a9d9a4617c66a3f383d WatchSource:0}: Error finding container 3f0f6f84b766e8e9c27ae5a1b15af9c9c97e80544f343a9d9a4617c66a3f383d: Status 404 returned error can't find the container with id 3f0f6f84b766e8e9c27ae5a1b15af9c9c97e80544f343a9d9a4617c66a3f383d Oct 31 17:26:48.938788 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.938767 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-862p4" Oct 31 17:26:48.948978 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:48.948936 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a53072_d4ff_45d1_ab01_249f63874e30.slice/crio-bfccff140157cc462720964a45b0913d94475aa4ea0a5bf5d4ea226da0da30ed WatchSource:0}: Error finding container bfccff140157cc462720964a45b0913d94475aa4ea0a5bf5d4ea226da0da30ed: Status 404 returned error can't find the container with id bfccff140157cc462720964a45b0913d94475aa4ea0a5bf5d4ea226da0da30ed Oct 31 17:26:48.961566 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.961538 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:26:48.971791 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.971769 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:26:48.972257 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:48.972232 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8b80aa_d5dc_41cd_8b89_5a45afc4064f.slice/crio-7a8936ba8a2ccb5380a9b3a6ceabd752ff7457d91566754d57429b10c95ced5c WatchSource:0}: Error finding container 7a8936ba8a2ccb5380a9b3a6ceabd752ff7457d91566754d57429b10c95ced5c: Status 404 returned error can't find the container with id 7a8936ba8a2ccb5380a9b3a6ceabd752ff7457d91566754d57429b10c95ced5c Oct 31 17:26:48.980688 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:48.980661 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e3309b_8a0f_42e7_bf62_a0d3d5a5b1a1.slice/crio-1f130503ff57c7c6901bc28e0fcaf54c7b3610b1fd438a136bd6f581d34752bc WatchSource:0}: Error finding container 1f130503ff57c7c6901bc28e0fcaf54c7b3610b1fd438a136bd6f581d34752bc: Status 404 returned error can't find the container with id 1f130503ff57c7c6901bc28e0fcaf54c7b3610b1fd438a136bd6f581d34752bc Oct 31 17:26:48.991547 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:48.991517 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:26:49.001294 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:26:49.001267 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3fc242_f912_4cdd_8f92_dcd8319ba40a.slice/crio-036583787fe6ee95d00c20e791023a15d74e43cd5fad9263dcb2c890f0372c01 WatchSource:0}: Error finding container 036583787fe6ee95d00c20e791023a15d74e43cd5fad9263dcb2c890f0372c01: Status 404 returned error can't find the container with id 036583787fe6ee95d00c20e791023a15d74e43cd5fad9263dcb2c890f0372c01 Oct 31 17:26:49.227996 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.227904 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:49.227996 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.227959 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:49.228240 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:49.228144 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:26:49.228240 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:49.228214 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs podName:d30a63cf-3116-466f-8e4d-c77f6eaacb37 nodeName:}" failed. No retries permitted until 2026-10-31 17:26:50.228193275 +0000 UTC m=+3.125024885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs") pod "network-metrics-daemon-9zfqw" (UID: "d30a63cf-3116-466f-8e4d-c77f6eaacb37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:26:49.228394 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:49.228286 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 17:26:49.228394 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:49.228341 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret podName:bec5b1da-4675-4713-9e51-9e7be00b5ba0 nodeName:}" failed. No retries permitted until 2026-10-31 17:26:50.228328576 +0000 UTC m=+3.125160180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret") pod "global-pull-secret-syncer-swbqv" (UID: "bec5b1da-4675-4713-9e51-9e7be00b5ba0") : object "kube-system"/"original-pull-secret" not registered Oct 31 17:26:49.271981 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.271941 2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 17:26:49.328790 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.328748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46kf\" (UniqueName: \"kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf\") pod \"network-check-target-cd9s8\" (UID: \"bcd594b6-86d2-47ff-bc81-03bd655d3954\") " pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:26:49.328983 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:49.328914 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 17:26:49.328983 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:49.328933 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 17:26:49.328983 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:49.328943 2566 projected.go:196] Error preparing data for projected volume kube-api-access-g46kf for pod openshift-network-diagnostics/network-check-target-cd9s8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:26:49.329156 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:49.329003 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf podName:bcd594b6-86d2-47ff-bc81-03bd655d3954 nodeName:}" failed. No retries permitted until 2026-10-31 17:26:50.328983552 +0000 UTC m=+3.225815155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g46kf" (UniqueName: "kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf") pod "network-check-target-cd9s8" (UID: "bcd594b6-86d2-47ff-bc81-03bd655d3954") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:26:49.596054 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.595932 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 17:21:49 +0000 UTC" deadline="2028-05-03 15:58:13.832415297 +0000 UTC" Oct 31 17:26:49.596054 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.596005 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13198h31m24.236414994s" Oct 31 17:26:49.644355 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.642672 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:49.644355 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:49.642824 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:26:49.644355 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.643266 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:49.644355 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:49.643399 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:26:49.680490 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.680420 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" event={"ID":"8765d5fdccb7ed0c5cd03e206e2ad979","Type":"ContainerStarted","Data":"49daaf0fda866993155a9e9a1e9d051c758c64ba9cde917ea68014e7b17ca97a"} Oct 31 17:26:49.688670 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.688585 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2klfh" event={"ID":"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f","Type":"ContainerStarted","Data":"7a8936ba8a2ccb5380a9b3a6ceabd752ff7457d91566754d57429b10c95ced5c"} Oct 31 17:26:49.708589 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.708551 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-58mtk" event={"ID":"159b2703-3b66-4ddc-8762-b4546b942282","Type":"ContainerStarted","Data":"3f0f6f84b766e8e9c27ae5a1b15af9c9c97e80544f343a9d9a4617c66a3f383d"} Oct 31 17:26:49.714691 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.714654 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" event={"ID":"44b0b68e-a3e0-4bb7-b987-92b7dacd0612","Type":"ContainerStarted","Data":"7c9f9bb4226272b3a084c5897c8a9bc45e7caf47230231e4ac98bd3aab9ba391"} Oct 31 17:26:49.733362 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.733319 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" event={"ID":"016409a4-3533-46e9-970d-4154787d4924","Type":"ContainerStarted","Data":"fc3b96f14c433569b9d1c4ad0849f0e7828dba0d2d8806b7fcb408977713ce73"} Oct 31 17:26:49.741449 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.741414 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" event={"ID":"8389bef8c4009e269aafe36de827cd22","Type":"ContainerStarted","Data":"d5989dbdb2d1c5d28a2b391e91e571a62d76150c1afe36bf4435b89aedd32b6c"} Oct 31 17:26:49.749954 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.749909 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" event={"ID":"7a3fc242-f912-4cdd-8f92-dcd8319ba40a","Type":"ContainerStarted","Data":"036583787fe6ee95d00c20e791023a15d74e43cd5fad9263dcb2c890f0372c01"} Oct 31 17:26:49.757811 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.757733 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cssnv" event={"ID":"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1","Type":"ContainerStarted","Data":"1f130503ff57c7c6901bc28e0fcaf54c7b3610b1fd438a136bd6f581d34752bc"} Oct 31 17:26:49.764871 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.764786 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-862p4" event={"ID":"64a53072-d4ff-45d1-ab01-249f63874e30","Type":"ContainerStarted","Data":"bfccff140157cc462720964a45b0913d94475aa4ea0a5bf5d4ea226da0da30ed"} Oct 31 17:26:49.771491 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.771402 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v9fs8" event={"ID":"76940cd0-ef63-4e10-965e-e871ab476281","Type":"ContainerStarted","Data":"94a1f08db33c8a171e666896062ed0eae9d3742fd533efb76c3c74ca5cc71816"} Oct 31 17:26:49.782524 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.782438 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9789n" event={"ID":"806f6a75-275f-48d2-ae95-03e1a5e46515","Type":"ContainerStarted","Data":"f2a476a670e5c7ff9a8ec1b564638865478650ecf8fd0f9596082da8b9bf5f69"} Oct 31 17:26:49.820770 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.820731 2566 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 17:26:49.860478 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:49.860397 2566 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 17:26:50.238117 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:50.238026 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:50.238117 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:50.238077 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:50.238376 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:50.238227 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:26:50.238376 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:50.238276 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs podName:d30a63cf-3116-466f-8e4d-c77f6eaacb37 nodeName:}" failed. No retries permitted until 2026-10-31 17:26:52.238262346 +0000 UTC m=+5.135093935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs") pod "network-metrics-daemon-9zfqw" (UID: "d30a63cf-3116-466f-8e4d-c77f6eaacb37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:26:50.238710 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:50.238691 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 17:26:50.238804 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:50.238749 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret podName:bec5b1da-4675-4713-9e51-9e7be00b5ba0 nodeName:}" failed. No retries permitted until 2026-10-31 17:26:52.238732878 +0000 UTC m=+5.135564475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret") pod "global-pull-secret-syncer-swbqv" (UID: "bec5b1da-4675-4713-9e51-9e7be00b5ba0") : object "kube-system"/"original-pull-secret" not registered Oct 31 17:26:50.339223 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:50.339182 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46kf\" (UniqueName: \"kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf\") pod \"network-check-target-cd9s8\" (UID: \"bcd594b6-86d2-47ff-bc81-03bd655d3954\") " pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:26:50.339431 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:50.339403 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 17:26:50.339431 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:50.339424 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 17:26:50.339551 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:50.339437 2566 projected.go:196] Error preparing data for projected volume kube-api-access-g46kf for pod openshift-network-diagnostics/network-check-target-cd9s8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:26:50.339551 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:50.339502 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf podName:bcd594b6-86d2-47ff-bc81-03bd655d3954 nodeName:}" failed. No retries permitted until 2026-10-31 17:26:52.339481204 +0000 UTC m=+5.236312796 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g46kf" (UniqueName: "kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf") pod "network-check-target-cd9s8" (UID: "bcd594b6-86d2-47ff-bc81-03bd655d3954") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:26:50.597300 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:50.597197 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 17:21:49 +0000 UTC" deadline="2028-04-13 21:10:07.200273098 +0000 UTC" Oct 31 17:26:50.597300 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:50.597248 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12723h43m16.603030052s" Oct 31 17:26:50.642391 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:50.642356 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:26:50.642566 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:50.642504 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:26:51.643625 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:51.643506 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:51.644315 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:51.643662 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:26:51.644315 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:51.644083 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:51.644315 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:51.644208 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:26:52.259804 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:52.259711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:52.259804 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:52.259774 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:52.260032 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:52.259951 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:26:52.260032 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:52.260020 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs podName:d30a63cf-3116-466f-8e4d-c77f6eaacb37 nodeName:}" failed. No retries permitted until 2026-10-31 17:26:56.259999944 +0000 UTC m=+9.156831551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs") pod "network-metrics-daemon-9zfqw" (UID: "d30a63cf-3116-466f-8e4d-c77f6eaacb37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:26:52.260505 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:52.260480 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 17:26:52.260583 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:52.260571 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret podName:bec5b1da-4675-4713-9e51-9e7be00b5ba0 nodeName:}" failed. No retries permitted until 2026-10-31 17:26:56.260533207 +0000 UTC m=+9.157364795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret") pod "global-pull-secret-syncer-swbqv" (UID: "bec5b1da-4675-4713-9e51-9e7be00b5ba0") : object "kube-system"/"original-pull-secret" not registered Oct 31 17:26:52.361063 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:52.361018 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46kf\" (UniqueName: \"kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf\") pod \"network-check-target-cd9s8\" (UID: \"bcd594b6-86d2-47ff-bc81-03bd655d3954\") " pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:26:52.361290 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:52.361269 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 17:26:52.361385 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:52.361312 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 17:26:52.361385 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:52.361328 2566 projected.go:196] Error preparing data for projected volume kube-api-access-g46kf for pod openshift-network-diagnostics/network-check-target-cd9s8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:26:52.361498 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:52.361390 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf podName:bcd594b6-86d2-47ff-bc81-03bd655d3954 nodeName:}" failed. No retries permitted until 2026-10-31 17:26:56.361371524 +0000 UTC m=+9.258203118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g46kf" (UniqueName: "kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf") pod "network-check-target-cd9s8" (UID: "bcd594b6-86d2-47ff-bc81-03bd655d3954") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:26:52.642496 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:52.642420 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:26:52.642655 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:52.642564 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:26:53.660189 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:53.659603 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:53.660189 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:53.659978 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:26:53.660189 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:53.660070 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:53.660189 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:53.660145 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:26:53.805712 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:53.805671 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" containerID="13b0da64d7d862384fad9d3d165ed348d769dcdf6f37794e66270033ee8e51e0" exitCode=0 Oct 31 17:26:53.806054 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:53.806020 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2klfh" event={"ID":"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f","Type":"ContainerDied","Data":"13b0da64d7d862384fad9d3d165ed348d769dcdf6f37794e66270033ee8e51e0"} Oct 31 17:26:53.814225 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:53.813592 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" event={"ID":"8389bef8c4009e269aafe36de827cd22","Type":"ContainerStarted","Data":"a7a445a65e3e2e610c575addbd63e9910a314bd935403af563a36d8982bb4947"} Oct 31 17:26:53.825963 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:53.825923 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v9fs8" event={"ID":"76940cd0-ef63-4e10-965e-e871ab476281","Type":"ContainerStarted","Data":"f95ebf5fa7f0de948481fe8692c6185c6ea6dca0a6429b41550c92f4f68bc0f9"} Oct 31 17:26:53.838719 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:53.838680 2566 generic.go:356] "Generic (PLEG): container finished" podID="8765d5fdccb7ed0c5cd03e206e2ad979" containerID="e8f1375616f561f3e093208f842939d4847fb5e18971a6d69a89bc3b3d753fe1" exitCode=0 Oct 31 17:26:53.838859 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:53.838733 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" event={"ID":"8765d5fdccb7ed0c5cd03e206e2ad979","Type":"ContainerDied","Data":"e8f1375616f561f3e093208f842939d4847fb5e18971a6d69a89bc3b3d753fe1"} Oct 31 17:26:53.968274 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:53.968141 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" podStartSLOduration=5.96812167 podStartE2EDuration="5.96812167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:26:53.967028239 +0000 UTC m=+6.863859957" watchObservedRunningTime="2026-10-31 17:26:53.96812167 +0000 UTC m=+6.864953285" Oct 31 17:26:54.643336 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:54.643300 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:26:54.643551 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:54.643444 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:26:54.846615 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:54.846550 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" event={"ID":"8765d5fdccb7ed0c5cd03e206e2ad979","Type":"ContainerStarted","Data":"d8d3cea8073073b0b0e25daa97b883b3f1c563309bff1ea1c8101e786556e371"} Oct 31 17:26:54.888942 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:54.888866 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v9fs8" podStartSLOduration=2.28590528 podStartE2EDuration="6.8888458s" totalImagesPullingTime="4.60294052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:26:54.013517654 +0000 UTC m=+6.910349268" watchObservedRunningTime="2026-10-31 17:26:54.8888458 +0000 UTC m=+7.785677413" Oct 31 17:26:54.889205 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:54.889149 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" podStartSLOduration=6.889137484 podStartE2EDuration="6.889137484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:26:54.888707273 +0000 UTC m=+7.785538887" watchObservedRunningTime="2026-10-31 17:26:54.889137484 +0000 UTC m=+7.785969101" Oct 31 17:26:55.661502 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:55.661469 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:55.661712 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:55.661510 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:55.661712 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:55.661655 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:26:55.661845 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:55.661717 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:26:55.855056 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:55.855015 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9789n" event={"ID":"806f6a75-275f-48d2-ae95-03e1a5e46515","Type":"ContainerStarted","Data":"1d296c4f8e7bd2a815384b1b154abbf35279ae560d9b066ec693c0906589fd20"} Oct 31 17:26:55.896233 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:55.896173 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9789n" podStartSLOduration=1.984359242 podStartE2EDuration="7.896156024s" totalImagesPullingTime="5.911796782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:26:55.896098854 +0000 UTC m=+8.792930467" watchObservedRunningTime="2026-10-31 17:26:55.896156024 +0000 UTC m=+8.792987640" Oct 31 17:26:56.301179 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:56.301059 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:56.301179 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:56.301118 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:56.301451 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:56.301287 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:26:56.301451 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:56.301396 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs podName:d30a63cf-3116-466f-8e4d-c77f6eaacb37 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:04.301352109 +0000 UTC m=+17.198183713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs") pod "network-metrics-daemon-9zfqw" (UID: "d30a63cf-3116-466f-8e4d-c77f6eaacb37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:26:56.301958 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:56.301916 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 17:26:56.302095 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:56.301977 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret podName:bec5b1da-4675-4713-9e51-9e7be00b5ba0 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:04.301960597 +0000 UTC m=+17.198792201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret") pod "global-pull-secret-syncer-swbqv" (UID: "bec5b1da-4675-4713-9e51-9e7be00b5ba0") : object "kube-system"/"original-pull-secret" not registered Oct 31 17:26:56.402888 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:56.402213 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46kf\" (UniqueName: \"kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf\") pod \"network-check-target-cd9s8\" (UID: \"bcd594b6-86d2-47ff-bc81-03bd655d3954\") " pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:26:56.402888 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:56.402421 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 17:26:56.402888 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:56.402443 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 17:26:56.402888 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:56.402456 2566 projected.go:196] Error preparing data for projected volume kube-api-access-g46kf for pod openshift-network-diagnostics/network-check-target-cd9s8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:26:56.402888 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:56.402516 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf podName:bcd594b6-86d2-47ff-bc81-03bd655d3954 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:04.402496764 +0000 UTC m=+17.299328371 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g46kf" (UniqueName: "kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf") pod "network-check-target-cd9s8" (UID: "bcd594b6-86d2-47ff-bc81-03bd655d3954") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:26:56.591174 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:56.590096 2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:26:56.591919 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:56.591663 2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:26:56.643276 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:56.642742 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:26:56.643276 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:56.642886 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:26:56.860219 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:56.859636 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" event={"ID":"016409a4-3533-46e9-970d-4154787d4924","Type":"ContainerStarted","Data":"fd92c4237c8fb5d7d10f64b1dbaed5b926e7ac703c011b70ae08c12d1e1a182e"} Oct 31 17:26:57.502931 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:57.502398 2566 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 17:26:57.601033 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:57.600865 2566 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T17:26:57.502418381Z","UUID":"559902e2-9026-430a-894d-5f7b23ad0b40","Handler":null,"Name":"","Endpoint":""} Oct 31 17:26:57.603015 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:57.602957 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 17:26:57.603015 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:57.602994 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 17:26:57.642796 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:57.642745 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:57.643007 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:57.642974 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:26:57.643476 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:57.643458 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:57.643619 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:57.643599 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:26:57.864914 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:57.864821 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" event={"ID":"016409a4-3533-46e9-970d-4154787d4924","Type":"ContainerStarted","Data":"3d3206437175defcbcce64cb7663c7c1b09e8549b74918d42c2e676ee913e4a9"} Oct 31 17:26:57.867401 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:57.867355 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-862p4" event={"ID":"64a53072-d4ff-45d1-ab01-249f63874e30","Type":"ContainerStarted","Data":"ee74c9f02a842c81d007edeb51f522e18ec8a76d328005b1d86b77ea6d6e6221"} Oct 31 17:26:58.642527 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:58.642491 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:26:58.642646 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:58.642624 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:26:58.873904 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:58.873632 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" event={"ID":"016409a4-3533-46e9-970d-4154787d4924","Type":"ContainerStarted","Data":"b1dd0320c969e77857cdd4e2116ccd7a2eeda68d771cbdfe7f1c31b20db8afe9"} Oct 31 17:26:58.876357 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:58.876315 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" containerID="2a8a0e6f3b3a7cc7c8ba024d862a25e7707045d45132d3afcbe49b89a1aa180b" exitCode=0 Oct 31 17:26:58.876531 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:58.876408 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2klfh" event={"ID":"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f","Type":"ContainerDied","Data":"2a8a0e6f3b3a7cc7c8ba024d862a25e7707045d45132d3afcbe49b89a1aa180b"} Oct 31 17:26:58.879201 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:58.879146 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-58mtk" event={"ID":"159b2703-3b66-4ddc-8762-b4546b942282","Type":"ContainerStarted","Data":"5e784c2b86f840e16ef9905923ebec319a6536c6d141e17e677e68025f6b501b"} Oct 31 17:26:58.916831 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:58.916723 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-862p4" podStartSLOduration=2.523868789 podStartE2EDuration="10.916704042s" totalImagesPullingTime="8.392835253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:26:57.91435321 +0000 UTC m=+10.811184825" watchObservedRunningTime="2026-10-31 17:26:58.916704042 +0000 UTC m=+11.813535654" Oct 31 17:26:58.917348 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:58.917309 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" podStartSLOduration=1.545844191 podStartE2EDuration="10.917299398s" totalImagesPullingTime="9.371455207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:26:58.916592818 +0000 UTC m=+11.813424427" watchObservedRunningTime="2026-10-31 17:26:58.917299398 +0000 UTC m=+11.814131008" Oct 31 17:26:58.999955 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:58.999897 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-58mtk" podStartSLOduration=1.82508472 podStartE2EDuration="10.999879224s" totalImagesPullingTime="9.174794504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:26:58.999539386 +0000 UTC m=+11.896370995" watchObservedRunningTime="2026-10-31 17:26:58.999879224 +0000 UTC m=+11.896710832" Oct 31 17:26:59.645844 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:59.645813 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:26:59.646010 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:59.645955 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:26:59.646369 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:59.646346 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:26:59.646508 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:26:59.646453 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:26:59.885171 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:59.885136 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" containerID="10fe61c8214d628e537283995b86a2e0685ff547968a13761602a99d96ac5b62" exitCode=0 Oct 31 17:26:59.885642 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:59.885220 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2klfh" event={"ID":"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f","Type":"ContainerDied","Data":"10fe61c8214d628e537283995b86a2e0685ff547968a13761602a99d96ac5b62"} Oct 31 17:26:59.891982 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:26:59.891951 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cssnv" event={"ID":"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1","Type":"ContainerStarted","Data":"dea4da58694d8bd298af946a3f26c950ad0a08b960a6d55f4dcd66298b00bc57"} Oct 31 17:27:00.642390 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:00.642362 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:00.642517 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:00.642474 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:00.897248 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:00.897156 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" containerID="96e8d1eefc7a13464ca873d4dc51d06d5304f325380796ac6c4a4d12991cbb8f" exitCode=0 Oct 31 17:27:00.897912 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:00.897271 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2klfh" event={"ID":"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f","Type":"ContainerDied","Data":"96e8d1eefc7a13464ca873d4dc51d06d5304f325380796ac6c4a4d12991cbb8f"} Oct 31 17:27:00.898861 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:00.898832 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" event={"ID":"44b0b68e-a3e0-4bb7-b987-92b7dacd0612","Type":"ContainerStarted","Data":"3d4ced0860d4a0fd4f8df2f4ca94197dddd630975facc500a582eb6a376c6918"} Oct 31 17:27:00.940358 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:00.940308 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cssnv" podStartSLOduration=4.245193289 podStartE2EDuration="12.940293374s" totalImagesPullingTime="8.695100085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:26:59.979627002 +0000 UTC m=+12.876458610" watchObservedRunningTime="2026-10-31 17:27:00.940293374 +0000 UTC m=+13.837124983" Oct 31 17:27:00.981021 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:00.980965 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" podStartSLOduration=1.7019619750000001 podStartE2EDuration="12.980951944s" totalImagesPullingTime="11.278989969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:00.980791082 +0000 UTC m=+13.877622693" watchObservedRunningTime="2026-10-31 17:27:00.980951944 +0000 UTC m=+13.877783552" Oct 31 17:27:01.642614 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:01.642583 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:01.642759 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:01.642693 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:01.643126 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:01.643105 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:01.643241 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:01.643223 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:02.643125 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:02.643085 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:02.643762 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:02.643204 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:02.907216 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:02.906914 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" containerID="d9365445890285d27c95b42868231e28b3d95ba734d7437b21dfd911e4fefb5f" exitCode=0 Oct 31 17:27:02.907216 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:02.907000 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2klfh" event={"ID":"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f","Type":"ContainerDied","Data":"d9365445890285d27c95b42868231e28b3d95ba734d7437b21dfd911e4fefb5f"} Oct 31 17:27:03.642618 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:03.642582 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:03.642787 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:03.642711 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:03.643230 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:03.643205 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:03.643617 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:03.643327 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:03.913021 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:03.912930 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" containerID="4b7ed9aceb2a4abb6ae92b1258af2be9354ce3315e431cb6ed7b33778b1aed35" exitCode=0 Oct 31 17:27:03.913021 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:03.912985 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2klfh" event={"ID":"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f","Type":"ContainerDied","Data":"4b7ed9aceb2a4abb6ae92b1258af2be9354ce3315e431cb6ed7b33778b1aed35"} Oct 31 17:27:04.365192 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.365101 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:04.365192 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.365140 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:04.365420 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:04.365248 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:27:04.365420 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:04.365278 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 17:27:04.365420 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:04.365321 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs podName:d30a63cf-3116-466f-8e4d-c77f6eaacb37 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:20.365307483 +0000 UTC m=+33.262139084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs") pod "network-metrics-daemon-9zfqw" (UID: "d30a63cf-3116-466f-8e4d-c77f6eaacb37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:27:04.365420 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:04.365336 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret podName:bec5b1da-4675-4713-9e51-9e7be00b5ba0 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:20.365328471 +0000 UTC m=+33.262160057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret") pod "global-pull-secret-syncer-swbqv" (UID: "bec5b1da-4675-4713-9e51-9e7be00b5ba0") : object "kube-system"/"original-pull-secret" not registered Oct 31 17:27:04.465722 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.465694 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46kf\" (UniqueName: \"kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf\") pod \"network-check-target-cd9s8\" (UID: \"bcd594b6-86d2-47ff-bc81-03bd655d3954\") " pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:04.465869 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:04.465812 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 17:27:04.465869 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:04.465827 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 17:27:04.465869 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:04.465836 2566 projected.go:196] Error preparing data for projected volume kube-api-access-g46kf for pod openshift-network-diagnostics/network-check-target-cd9s8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:27:04.465952 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:04.465895 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf podName:bcd594b6-86d2-47ff-bc81-03bd655d3954 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:20.465883517 +0000 UTC m=+33.362715104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g46kf" (UniqueName: "kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf") pod "network-check-target-cd9s8" (UID: "bcd594b6-86d2-47ff-bc81-03bd655d3954") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:27:04.642979 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.642959 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:04.643099 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:04.643079 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:04.919730 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.919471 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2klfh" event={"ID":"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f","Type":"ContainerStarted","Data":"982f5e84b9ba5feee725374f3a9d1ea85e16dec2e1f6a176b0da3c9445f81c70"} Oct 31 17:27:04.922467 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.922442 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:27:04.922844 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.922817 2566 generic.go:356] "Generic (PLEG): container finished" podID="7a3fc242-f912-4cdd-8f92-dcd8319ba40a" containerID="89272db701614577b68dff17143a65604ce33b86a053ab6084beee0095c46dab" exitCode=1 Oct 31 17:27:04.922913 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.922873 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" event={"ID":"7a3fc242-f912-4cdd-8f92-dcd8319ba40a","Type":"ContainerStarted","Data":"442d20efe0820b9612359426e344b5d288947ad0109236245a81261dfb412e0a"} Oct 31 17:27:04.922967 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.922953 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" event={"ID":"7a3fc242-f912-4cdd-8f92-dcd8319ba40a","Type":"ContainerStarted","Data":"fdb3edd04f8213267c22650cc5034b4c89e2654bb5ad44a26fdafccc3cb46458"} Oct 31 17:27:04.923016 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.923004 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" event={"ID":"7a3fc242-f912-4cdd-8f92-dcd8319ba40a","Type":"ContainerStarted","Data":"f2576e05fb56dd599df434720608e3d1ce99e39820bdb4cd6544fe973cbe6ffa"} Oct 31 17:27:04.923066 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.923053 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" event={"ID":"7a3fc242-f912-4cdd-8f92-dcd8319ba40a","Type":"ContainerStarted","Data":"bae9bfec5bf973858155ad276c7a3a4bf7e9667e74e840dcd0bba1b2edd7825b"} Oct 31 17:27:04.923153 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.923140 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" event={"ID":"7a3fc242-f912-4cdd-8f92-dcd8319ba40a","Type":"ContainerDied","Data":"89272db701614577b68dff17143a65604ce33b86a053ab6084beee0095c46dab"} Oct 31 17:27:04.923210 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.923197 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" event={"ID":"7a3fc242-f912-4cdd-8f92-dcd8319ba40a","Type":"ContainerStarted","Data":"78afc1b7167136c958f4df1444a8c969075a5ddd6447d420237d970e26ddc2d8"} Oct 31 17:27:04.963660 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:04.963608 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2klfh" podStartSLOduration=6.187425403 podStartE2EDuration="16.963595024s" totalImagesPullingTime="10.776169621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:04.963188079 +0000 UTC m=+17.860019690" watchObservedRunningTime="2026-10-31 17:27:04.963595024 +0000 UTC m=+17.860426632" Oct 31 17:27:05.644955 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:05.644922 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:05.645098 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:05.644922 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:05.645098 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:05.645050 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:05.645098 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:05.645088 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:06.642414 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:06.642385 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:06.642776 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:06.642480 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:07.643396 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:07.643354 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:07.643874 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:07.643475 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:07.643874 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:07.643522 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:07.643874 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:07.643611 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:07.933304 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:07.933215 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:27:07.933622 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:07.933600 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" event={"ID":"7a3fc242-f912-4cdd-8f92-dcd8319ba40a","Type":"ContainerStarted","Data":"c2eddca2e121302d9df1254a69baacb7518893603cf70ee498b0405c261e89a7"} Oct 31 17:27:08.643296 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:08.643225 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:08.643488 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:08.643358 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:09.642616 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:09.642579 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:09.642788 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:09.642579 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:09.642788 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:09.642695 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:09.642788 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:09.642755 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:09.941682 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:09.941614 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:27:09.942336 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:09.941964 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" event={"ID":"7a3fc242-f912-4cdd-8f92-dcd8319ba40a","Type":"ContainerStarted","Data":"d43ce9140245533086c1b2e85337812dab63fa26cc501888d56daffeac892408"} Oct 31 17:27:09.942336 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:09.942280 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:09.942442 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:09.942402 2566 scope.go:122] "RemoveContainer" containerID="89272db701614577b68dff17143a65604ce33b86a053ab6084beee0095c46dab" Oct 31 17:27:09.958672 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:09.958644 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:10.424924 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:10.424892 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:27:10.425076 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:10.425062 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 17:27:10.425660 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:10.425637 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:27:10.642854 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:10.642819 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:10.643009 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:10.642919 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:10.948102 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:10.948076 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:27:10.948596 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:10.948469 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" event={"ID":"7a3fc242-f912-4cdd-8f92-dcd8319ba40a","Type":"ContainerStarted","Data":"64432037cacc99c214946de8665a06a98e5c497be819cdd2a8c4d29a63a877af"} Oct 31 17:27:10.948596 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:10.948555 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 17:27:10.948760 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:10.948740 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:10.963615 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:10.963583 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:11.022662 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:11.022609 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" podStartSLOduration=7.617504363 podStartE2EDuration="23.022593869s" totalImagesPullingTime="15.405089506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:11.022026866 +0000 UTC m=+23.918858513" watchObservedRunningTime="2026-10-31 17:27:11.022593869 +0000 UTC m=+23.919425554" Oct 31 17:27:11.642923 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:11.642878 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:11.642923 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:11.642903 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:11.643170 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:11.643008 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:11.643170 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:11.643151 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:11.951080 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:11.951006 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 17:27:12.642304 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:12.642244 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:12.642473 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:12.642389 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:12.953408 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:12.953333 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 17:27:13.579110 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:13.579083 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-58mtk_159b2703-3b66-4ddc-8762-b4546b942282/dns-node-resolver/0.log" Oct 31 17:27:13.642708 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:13.642680 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:13.642849 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:13.642717 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:13.642849 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:13.642812 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:13.642947 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:13.642908 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:14.526358 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:14.526328 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9789n_806f6a75-275f-48d2-ae95-03e1a5e46515/node-ca/0.log" Oct 31 17:27:14.643240 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:14.643201 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:14.643429 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:14.643330 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:15.642820 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:15.642632 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:15.643458 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:15.642704 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:15.643458 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:15.642909 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:15.643458 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:15.642965 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:16.642700 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:16.642665 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:16.642873 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:16.642774 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:16.785869 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:16.785830 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:16.786063 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:16.786050 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 17:27:16.796384 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:16.796342 2566 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" podUID="7a3fc242-f912-4cdd-8f92-dcd8319ba40a" containerName="ovnkube-controller" probeResult="failure" output="" Oct 31 17:27:16.805029 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:16.804996 2566 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" podUID="7a3fc242-f912-4cdd-8f92-dcd8319ba40a" containerName="ovnkube-controller" probeResult="failure" output="" Oct 31 17:27:17.612053 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:17.612011 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v9fs8_76940cd0-ef63-4e10-965e-e871ab476281/konnectivity-agent/0.log" limit=1024 Oct 31 17:27:17.643812 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:17.643778 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:17.644278 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:17.643891 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:17.644278 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:17.643970 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:17.644278 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:17.644049 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:17.752388 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:17.752337 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v9fs8_76940cd0-ef63-4e10-965e-e871ab476281/konnectivity-agent/0.log" limit=1024 Oct 31 17:27:17.877533 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:17.877452 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v9fs8_76940cd0-ef63-4e10-965e-e871ab476281/konnectivity-agent/0.log" limit=1024 Oct 31 17:27:18.000891 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:18.000856 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v9fs8_76940cd0-ef63-4e10-965e-e871ab476281/konnectivity-agent/0.log" limit=1024 Oct 31 17:27:18.163577 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:18.163549 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v9fs8_76940cd0-ef63-4e10-965e-e871ab476281/konnectivity-agent/0.log" limit=1024 Oct 31 17:27:18.373612 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:18.373580 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v9fs8_76940cd0-ef63-4e10-965e-e871ab476281/konnectivity-agent/0.log" limit=1024 Oct 31 17:27:18.642678 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:18.642653 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:18.642834 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:18.642757 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:19.642643 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:19.642602 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:19.643052 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:19.642770 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:19.643052 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:19.642800 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:19.643052 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:19.642894 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:20.373671 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:20.373637 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:20.373671 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:20.373671 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:20.373969 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:20.373776 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:27:20.373969 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:20.373782 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 17:27:20.373969 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:20.373843 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret podName:bec5b1da-4675-4713-9e51-9e7be00b5ba0 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:52.373823858 +0000 UTC m=+65.270655458 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret") pod "global-pull-secret-syncer-swbqv" (UID: "bec5b1da-4675-4713-9e51-9e7be00b5ba0") : object "kube-system"/"original-pull-secret" not registered Oct 31 17:27:20.373969 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:20.373864 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs podName:d30a63cf-3116-466f-8e4d-c77f6eaacb37 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:52.373854068 +0000 UTC m=+65.270685655 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs") pod "network-metrics-daemon-9zfqw" (UID: "d30a63cf-3116-466f-8e4d-c77f6eaacb37") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 17:27:20.474073 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:20.474032 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46kf\" (UniqueName: \"kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf\") pod \"network-check-target-cd9s8\" (UID: \"bcd594b6-86d2-47ff-bc81-03bd655d3954\") " pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:20.474225 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:20.474158 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 17:27:20.474225 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:20.474172 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 17:27:20.474225 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:20.474181 2566 projected.go:196] Error preparing data for projected volume kube-api-access-g46kf for pod openshift-network-diagnostics/network-check-target-cd9s8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:27:20.474225 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:20.474227 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf podName:bcd594b6-86d2-47ff-bc81-03bd655d3954 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:52.474214692 +0000 UTC m=+65.371046292 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g46kf" (UniqueName: "kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf") pod "network-check-target-cd9s8" (UID: "bcd594b6-86d2-47ff-bc81-03bd655d3954") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 17:27:20.643289 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:20.643247 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:20.643666 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:20.643365 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:21.643033 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:21.642995 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:21.643222 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:21.643117 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:21.643222 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:21.643173 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:21.643594 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:21.643305 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:22.642328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:22.642292 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:22.642506 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:22.642391 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:23.642419 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:23.642388 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:23.642419 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:23.642418 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:23.642858 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:23.642489 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:23.642858 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:23.642609 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:24.642229 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:24.642202 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:24.642410 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:24.642308 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:25.642525 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:25.642493 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:25.642930 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:25.642542 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:25.642930 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:25.642622 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:25.642930 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:25.642660 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:26.642506 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:26.642469 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:26.642679 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:26.642571 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:27.643758 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:27.643728 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:27.644154 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:27.643825 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:27.644154 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:27.643877 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:27.644154 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:27.643943 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:28.642509 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:28.642475 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:28.642683 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:28.642576 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:29.643379 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:29.643345 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:29.643379 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:29.643372 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:29.643865 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:29.643446 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:29.643865 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:29.643591 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:30.643044 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:30.643017 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:30.643213 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:30.643104 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:31.026595 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:31.026558 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-swbqv"] Oct 31 17:27:31.027151 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:31.026680 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:31.027151 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:31.026797 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:31.030302 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:31.030270 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cd9s8"] Oct 31 17:27:31.030451 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:31.030384 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:31.030521 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:31.030491 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:31.031083 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:31.031050 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9zfqw"] Oct 31 17:27:31.031160 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:31.031148 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:31.031292 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:31.031271 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:32.158294 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:32.158270 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-58mtk_159b2703-3b66-4ddc-8762-b4546b942282/dns-node-resolver/0.log" Oct 31 17:27:32.642528 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:32.642491 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:32.642702 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:32.642525 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:32.642702 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:32.642594 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:32.642702 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:32.642642 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:32.642875 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:32.642761 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:32.642875 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:32.642847 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:33.357933 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:33.357905 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9789n_806f6a75-275f-48d2-ae95-03e1a5e46515/node-ca/0.log" Oct 31 17:27:34.643011 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:34.642818 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:34.643450 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:34.642819 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:34.643450 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:34.643093 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9zfqw" podUID="d30a63cf-3116-466f-8e4d-c77f6eaacb37" Oct 31 17:27:34.643450 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:34.642866 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:34.643450 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:34.643149 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd9s8" podUID="bcd594b6-86d2-47ff-bc81-03bd655d3954" Oct 31 17:27:34.643450 ip-10-0-139-107 kubenswrapper[2566]: E1031 17:27:34.643231 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swbqv" podUID="bec5b1da-4675-4713-9e51-9e7be00b5ba0" Oct 31 17:27:35.809656 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.809633 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeReady" Oct 31 17:27:35.809965 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.809747 2566 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 17:27:35.863420 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.863385 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4jjks"] Oct 31 17:27:35.865482 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.865462 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:35.867954 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.867932 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x8mnz"] Oct 31 17:27:35.870073 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.870056 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x8mnz" Oct 31 17:27:35.883285 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.883240 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-knkx9"] Oct 31 17:27:35.885323 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.885300 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:35.886031 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.886006 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 17:27:35.886148 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.886132 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4g54t\"" Oct 31 17:27:35.886969 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.886911 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4jjks"] Oct 31 17:27:35.889071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.889051 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x8mnz"] Oct 31 17:27:35.909383 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.909350 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-knkx9"] Oct 31 17:27:35.965434 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.965398 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 17:27:35.965434 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.965420 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 17:27:35.965434 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.965434 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 17:27:35.965679 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.965437 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 17:27:35.965679 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.965452 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 17:27:35.965679 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.965509 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 17:27:35.979037 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.979009 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdphq\" (UniqueName: \"kubernetes.io/projected/6fa107ff-c9d8-41a8-9baa-190bd8714606-kube-api-access-rdphq\") pod \"ingress-canary-x8mnz\" (UID: \"6fa107ff-c9d8-41a8-9baa-190bd8714606\") " pod="openshift-ingress-canary/ingress-canary-x8mnz" Oct 31 17:27:35.979087 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.979048 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9tw2\" (UniqueName: \"kubernetes.io/projected/795fdffb-4631-44b0-a0fa-91b74548f1e9-kube-api-access-w9tw2\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:35.979087 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.979074 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/795fdffb-4631-44b0-a0fa-91b74548f1e9-crio-socket\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:35.979197 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.979170 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-metrics-tls\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:35.979234 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.979205 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7s8n\" (UniqueName: \"kubernetes.io/projected/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-kube-api-access-x7s8n\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:35.979234 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.979224 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795fdffb-4631-44b0-a0fa-91b74548f1e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:35.979337 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.979241 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fa107ff-c9d8-41a8-9baa-190bd8714606-cert\") pod \"ingress-canary-x8mnz\" (UID: \"6fa107ff-c9d8-41a8-9baa-190bd8714606\") " pod="openshift-ingress-canary/ingress-canary-x8mnz" Oct 31 17:27:35.979371 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.979334 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-config-volume\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:35.979403 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.979369 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/795fdffb-4631-44b0-a0fa-91b74548f1e9-data-volume\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:35.979455 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.979439 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-tmp-dir\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:35.979487 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.979468 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795fdffb-4631-44b0-a0fa-91b74548f1e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:35.986136 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.986113 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8gjvp\"" Oct 31 17:27:35.986178 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.986113 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 17:27:35.986178 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.986166 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 17:27:35.986234 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:35.986177 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dm9tc\"" Oct 31 17:27:36.080809 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.080722 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/795fdffb-4631-44b0-a0fa-91b74548f1e9-crio-socket\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:36.080809 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.080763 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-metrics-tls\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:36.080809 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.080782 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s8n\" (UniqueName: \"kubernetes.io/projected/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-kube-api-access-x7s8n\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:36.080809 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.080805 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795fdffb-4631-44b0-a0fa-91b74548f1e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:36.081124 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.080830 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fa107ff-c9d8-41a8-9baa-190bd8714606-cert\") pod \"ingress-canary-x8mnz\" (UID: \"6fa107ff-c9d8-41a8-9baa-190bd8714606\") " pod="openshift-ingress-canary/ingress-canary-x8mnz" Oct 31 17:27:36.081124 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.080967 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/795fdffb-4631-44b0-a0fa-91b74548f1e9-crio-socket\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:36.081124 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.081005 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-config-volume\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:36.081124 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.081040 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/795fdffb-4631-44b0-a0fa-91b74548f1e9-data-volume\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:36.081343 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.081186 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-tmp-dir\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:36.081343 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.081229 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795fdffb-4631-44b0-a0fa-91b74548f1e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:36.081343 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.081292 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdphq\" (UniqueName: \"kubernetes.io/projected/6fa107ff-c9d8-41a8-9baa-190bd8714606-kube-api-access-rdphq\") pod \"ingress-canary-x8mnz\" (UID: \"6fa107ff-c9d8-41a8-9baa-190bd8714606\") " pod="openshift-ingress-canary/ingress-canary-x8mnz" Oct 31 17:27:36.081343 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.081322 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9tw2\" (UniqueName: \"kubernetes.io/projected/795fdffb-4631-44b0-a0fa-91b74548f1e9-kube-api-access-w9tw2\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:36.081532 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.081355 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/795fdffb-4631-44b0-a0fa-91b74548f1e9-data-volume\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:36.081590 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.081569 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-tmp-dir\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:36.081652 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.081634 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-config-volume\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:36.081652 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.081640 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795fdffb-4631-44b0-a0fa-91b74548f1e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:36.084767 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.084738 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795fdffb-4631-44b0-a0fa-91b74548f1e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:36.084862 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.084750 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-metrics-tls\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:36.085564 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.085218 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fa107ff-c9d8-41a8-9baa-190bd8714606-cert\") pod \"ingress-canary-x8mnz\" (UID: \"6fa107ff-c9d8-41a8-9baa-190bd8714606\") " pod="openshift-ingress-canary/ingress-canary-x8mnz" Oct 31 17:27:36.100766 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.100733 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7s8n\" (UniqueName: \"kubernetes.io/projected/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-kube-api-access-x7s8n\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:36.100934 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.100916 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9tw2\" (UniqueName: \"kubernetes.io/projected/795fdffb-4631-44b0-a0fa-91b74548f1e9-kube-api-access-w9tw2\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:36.101033 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.101017 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdphq\" (UniqueName: \"kubernetes.io/projected/6fa107ff-c9d8-41a8-9baa-190bd8714606-kube-api-access-rdphq\") pod \"ingress-canary-x8mnz\" (UID: \"6fa107ff-c9d8-41a8-9baa-190bd8714606\") " pod="openshift-ingress-canary/ingress-canary-x8mnz" Oct 31 17:27:36.179052 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.179013 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:36.183137 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.183119 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x8mnz" Oct 31 17:27:36.198525 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.198500 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:36.490353 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.490278 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4jjks"] Oct 31 17:27:36.493377 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:36.493352 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd29fa4f_f33c_4c02_988c_afcfb47b6ca0.slice/crio-07759b19aa29bec829ba00205faa69eaa4c5b1226f243986267cda739eb23d61 WatchSource:0}: Error finding container 07759b19aa29bec829ba00205faa69eaa4c5b1226f243986267cda739eb23d61: Status 404 returned error can't find the container with id 07759b19aa29bec829ba00205faa69eaa4c5b1226f243986267cda739eb23d61 Oct 31 17:27:36.504345 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.504323 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x8mnz"] Oct 31 17:27:36.507176 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:36.507153 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa107ff_c9d8_41a8_9baa_190bd8714606.slice/crio-6bf9846392b4c9d13b4427ecf3a61853a69972746745fbe33807869789fcc421 WatchSource:0}: Error finding container 6bf9846392b4c9d13b4427ecf3a61853a69972746745fbe33807869789fcc421: Status 404 returned error can't find the container with id 6bf9846392b4c9d13b4427ecf3a61853a69972746745fbe33807869789fcc421 Oct 31 17:27:36.515757 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.515736 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-knkx9"] Oct 31 17:27:36.518520 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:36.518497 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795fdffb_4631_44b0_a0fa_91b74548f1e9.slice/crio-7c96ab1a6b8a6e25808f08deafcc74ada17b43b7b6f6a3c94b620f66b7052a79 WatchSource:0}: Error finding container 7c96ab1a6b8a6e25808f08deafcc74ada17b43b7b6f6a3c94b620f66b7052a79: Status 404 returned error can't find the container with id 7c96ab1a6b8a6e25808f08deafcc74ada17b43b7b6f6a3c94b620f66b7052a79 Oct 31 17:27:36.642669 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.642634 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:36.642669 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.642665 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:36.642863 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.642675 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:36.659156 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.659104 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 17:27:36.659306 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.659194 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nk5tf\"" Oct 31 17:27:36.659306 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.659215 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 17:27:36.659306 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.659297 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 17:27:36.659460 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.659347 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tj66q\"" Oct 31 17:27:36.659510 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.659458 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 17:27:36.659510 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:36.659496 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z2hdt\"" Oct 31 17:27:37.007330 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:37.007288 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4jjks" event={"ID":"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0","Type":"ContainerStarted","Data":"07759b19aa29bec829ba00205faa69eaa4c5b1226f243986267cda739eb23d61"} Oct 31 17:27:37.009955 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:37.009922 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-knkx9" event={"ID":"795fdffb-4631-44b0-a0fa-91b74548f1e9","Type":"ContainerStarted","Data":"767848d28ddabcc0a7a8b6f852963100132c2f1a6bc2fa25ca798da16bc9944a"} Oct 31 17:27:37.010095 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:37.009984 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-knkx9" event={"ID":"795fdffb-4631-44b0-a0fa-91b74548f1e9","Type":"ContainerStarted","Data":"7c96ab1a6b8a6e25808f08deafcc74ada17b43b7b6f6a3c94b620f66b7052a79"} Oct 31 17:27:37.011423 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:37.011396 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x8mnz" event={"ID":"6fa107ff-c9d8-41a8-9baa-190bd8714606","Type":"ContainerStarted","Data":"6bf9846392b4c9d13b4427ecf3a61853a69972746745fbe33807869789fcc421"} Oct 31 17:27:38.017609 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.017573 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-knkx9" event={"ID":"795fdffb-4631-44b0-a0fa-91b74548f1e9","Type":"ContainerStarted","Data":"b749ab2e13b771e017bfcc4d18668f6441aac765d622e6f0cda217795e16d39a"} Oct 31 17:27:38.073879 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.073841 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv"] Oct 31 17:27:38.077053 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.076718 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.098290 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.098221 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 17:27:38.098829 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.098724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e895668-db69-4458-9e3d-fcd82e31084b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.098829 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.098779 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nkrt\" (UniqueName: \"kubernetes.io/projected/6e895668-db69-4458-9e3d-fcd82e31084b-kube-api-access-8nkrt\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.099010 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.098937 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e895668-db69-4458-9e3d-fcd82e31084b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.099077 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.099008 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e895668-db69-4458-9e3d-fcd82e31084b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.113593 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.113518 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5hshp\"" Oct 31 17:27:38.113593 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.113533 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 17:27:38.113812 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.113622 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 17:27:38.113812 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.113705 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 17:27:38.113812 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.113796 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 17:27:38.121784 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.115219 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv"] Oct 31 17:27:38.136321 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.136291 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf"] Oct 31 17:27:38.139094 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.139063 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-66bdk"] Oct 31 17:27:38.139242 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.139198 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.143269 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.143225 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.161772 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.161740 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf"] Oct 31 17:27:38.163513 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.163482 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mcfpr\"" Oct 31 17:27:38.163619 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.163515 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 17:27:38.163720 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.163705 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 17:27:38.163762 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.163739 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 17:27:38.163965 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.163923 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 17:27:38.163965 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.163937 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4pmd9\"" Oct 31 17:27:38.196323 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.196294 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 17:27:38.196482 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.196436 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 17:27:38.199945 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.199911 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-root\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.200082 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.199981 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.200082 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200012 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.200082 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200043 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.200082 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200071 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.200307 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200096 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.200307 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200121 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93b4b2af-0bd0-4907-9535-0ce4ef46383c-metrics-client-ca\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.200307 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200144 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-tls\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.200307 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200171 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6wh\" (UniqueName: \"kubernetes.io/projected/93b4b2af-0bd0-4907-9535-0ce4ef46383c-kube-api-access-hh6wh\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.200307 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200199 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-textfile\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.200307 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200232 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e895668-db69-4458-9e3d-fcd82e31084b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.200307 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200278 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-sys\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.200532 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200315 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx2wp\" (UniqueName: \"kubernetes.io/projected/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-api-access-jx2wp\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.200532 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200350 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-wtmp\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.200532 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200376 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e895668-db69-4458-9e3d-fcd82e31084b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.200532 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200397 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.200532 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200416 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e895668-db69-4458-9e3d-fcd82e31084b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.200532 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200434 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkrt\" (UniqueName: \"kubernetes.io/projected/6e895668-db69-4458-9e3d-fcd82e31084b-kube-api-access-8nkrt\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.200532 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.200451 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-accelerators-collector-config\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.201811 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.201777 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e895668-db69-4458-9e3d-fcd82e31084b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.204346 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.204320 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e895668-db69-4458-9e3d-fcd82e31084b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.204710 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.204686 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e895668-db69-4458-9e3d-fcd82e31084b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.222958 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.222925 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkrt\" (UniqueName: \"kubernetes.io/projected/6e895668-db69-4458-9e3d-fcd82e31084b-kube-api-access-8nkrt\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.301218 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301187 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-wtmp\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.301361 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301235 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.301361 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301287 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-accelerators-collector-config\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.301361 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301316 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-root\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.301361 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.301550 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301376 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.301550 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301396 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.301550 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301413 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.301550 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301410 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-wtmp\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.301550 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301431 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.301550 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301473 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93b4b2af-0bd0-4907-9535-0ce4ef46383c-metrics-client-ca\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.301550 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301501 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-tls\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.301550 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301532 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6wh\" (UniqueName: \"kubernetes.io/projected/93b4b2af-0bd0-4907-9535-0ce4ef46383c-kube-api-access-hh6wh\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.301909 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301566 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-textfile\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.301909 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301597 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-sys\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.301909 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.301636 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx2wp\" (UniqueName: \"kubernetes.io/projected/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-api-access-jx2wp\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.302609 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.302231 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-textfile\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.302609 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.302302 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-sys\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.302609 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.302470 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93b4b2af-0bd0-4907-9535-0ce4ef46383c-metrics-client-ca\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.303135 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.303111 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-accelerators-collector-config\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.303232 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.303207 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.303328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.303309 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-root\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.303458 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.303438 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.303959 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.303934 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.304592 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.304566 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.305843 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.305815 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-tls\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.306112 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.306089 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.306791 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.306758 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.323481 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.323450 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx2wp\" (UniqueName: \"kubernetes.io/projected/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-api-access-jx2wp\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.324921 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.324900 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6wh\" (UniqueName: \"kubernetes.io/projected/93b4b2af-0bd0-4907-9535-0ce4ef46383c-kube-api-access-hh6wh\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.395728 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.395700 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:38.459279 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.459030 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:38.472001 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.468408 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:38.719458 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.719422 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv"] Oct 31 17:27:38.723452 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:38.723417 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e895668_db69_4458_9e3d_fcd82e31084b.slice/crio-e724a097ca03b771375c87e4ac2eea048be276d8095f4b7dc5d4e14fb980c9e6 WatchSource:0}: Error finding container e724a097ca03b771375c87e4ac2eea048be276d8095f4b7dc5d4e14fb980c9e6: Status 404 returned error can't find the container with id e724a097ca03b771375c87e4ac2eea048be276d8095f4b7dc5d4e14fb980c9e6 Oct 31 17:27:38.769218 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:38.769183 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf"] Oct 31 17:27:38.773113 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:38.773060 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9206f5fc_3e4d_4574_8c55_aaca1a33e3d4.slice/crio-871100da38ff67712afa26e098f88567bdb9f934004f9067c173a20a7adcb399 WatchSource:0}: Error finding container 871100da38ff67712afa26e098f88567bdb9f934004f9067c173a20a7adcb399: Status 404 returned error can't find the container with id 871100da38ff67712afa26e098f88567bdb9f934004f9067c173a20a7adcb399 Oct 31 17:27:39.023427 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.023284 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66bdk" event={"ID":"93b4b2af-0bd0-4907-9535-0ce4ef46383c","Type":"ContainerStarted","Data":"010c48d00c6359c0bfc73132137eda85a54cfaf74a110583fbe5d4583dc97366"} Oct 31 17:27:39.024677 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.024636 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" event={"ID":"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4","Type":"ContainerStarted","Data":"871100da38ff67712afa26e098f88567bdb9f934004f9067c173a20a7adcb399"} Oct 31 17:27:39.026997 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.026873 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" event={"ID":"6e895668-db69-4458-9e3d-fcd82e31084b","Type":"ContainerStarted","Data":"7eeeced044ac2b91f8d932093098026f34e2f4a0e6049961694fdf30198780af"} Oct 31 17:27:39.026997 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.026928 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" event={"ID":"6e895668-db69-4458-9e3d-fcd82e31084b","Type":"ContainerStarted","Data":"da6346abfdbed533456858eac67e473307f822ace85ba062f35ac9e732215126"} Oct 31 17:27:39.026997 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.026954 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" event={"ID":"6e895668-db69-4458-9e3d-fcd82e31084b","Type":"ContainerStarted","Data":"e724a097ca03b771375c87e4ac2eea048be276d8095f4b7dc5d4e14fb980c9e6"} Oct 31 17:27:39.029510 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.029481 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4jjks" event={"ID":"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0","Type":"ContainerStarted","Data":"6d3de6401d2a623179d2e58a1ff964647d725efc1a28451bd02a934eafc78db0"} Oct 31 17:27:39.029628 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.029536 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4jjks" event={"ID":"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0","Type":"ContainerStarted","Data":"14963ef71244b85bd674c93044fb12b55d30b93b101e21a710887cad90f29662"} Oct 31 17:27:39.029628 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.029592 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:39.031984 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.031956 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-knkx9" event={"ID":"795fdffb-4631-44b0-a0fa-91b74548f1e9","Type":"ContainerStarted","Data":"b8cdc04231af7b361493f64639dca6ffc9f77f8242710ae900919d74b6438e96"} Oct 31 17:27:39.074484 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.074431 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-4jjks" podStartSLOduration=2.274086521 podStartE2EDuration="4.074390232s" totalImagesPullingTime="1.800303711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:39.072397677 +0000 UTC m=+51.969229288" watchObservedRunningTime="2026-10-31 17:27:39.074390232 +0000 UTC m=+51.971221841" Oct 31 17:27:39.117190 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.117153 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f77577466-bjlt5"] Oct 31 17:27:39.119818 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.119790 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.141005 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.140925 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 17:27:39.141005 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.140955 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 17:27:39.141223 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.141106 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 17:27:39.141631 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.141368 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 17:27:39.141631 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.141403 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 17:27:39.141631 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.141478 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 17:27:39.141631 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.141587 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ht46x\"" Oct 31 17:27:39.141903 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.141706 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 17:27:39.142379 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.142334 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-knkx9" podStartSLOduration=1.930865866 podStartE2EDuration="4.142317921s" totalImagesPullingTime="2.211452055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:27:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:39.119773957 +0000 UTC m=+52.016605565" watchObservedRunningTime="2026-10-31 17:27:39.142317921 +0000 UTC m=+52.039149531" Oct 31 17:27:39.143478 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.143457 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f77577466-bjlt5"] Oct 31 17:27:39.155579 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.155551 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 17:27:39.210046 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.210005 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.210046 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.210043 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.210324 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.210074 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.210324 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.210283 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.210429 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.210328 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.210429 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.210357 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.210429 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.210386 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/511ee639-1612-4e36-812e-70f7865291c1-tmp\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.210576 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.210443 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.310939 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.310847 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.310939 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.310903 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.310939 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.310934 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.311183 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.310964 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/511ee639-1612-4e36-812e-70f7865291c1-tmp\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.311183 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.311020 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.311183 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.311054 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.311183 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.311080 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.311183 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.311098 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.311867 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.311834 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.314957 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.312506 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/511ee639-1612-4e36-812e-70f7865291c1-tmp\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.314957 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.313019 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.314957 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.314331 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.314957 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.314903 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.315827 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.315807 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.318943 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.317817 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.334545 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.334476 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.432710 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.432664 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:39.582960 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.581492 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 17:27:39.586209 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.585890 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.603328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.602602 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 17:27:39.603328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.602696 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hzhgj\"" Oct 31 17:27:39.603328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.602779 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 17:27:39.603328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.602842 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 17:27:39.603328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.602961 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 17:27:39.603328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.603025 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 17:27:39.603328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.603025 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 17:27:39.603328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.603196 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 17:27:39.604315 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.604262 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 17:27:39.616370 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.616342 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 17:27:39.617474 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617031 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2mw5\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-kube-api-access-q2mw5\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.617474 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617078 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.617474 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617105 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.617474 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617134 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.617474 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617336 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.617474 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617382 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.617474 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617416 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.617474 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617468 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.617954 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617548 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.617954 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617576 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-config-volume\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.617954 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617595 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-config-out\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.617954 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617614 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-web-config\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.617954 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.617643 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.702531 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.702479 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 17:27:39.719444 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.718913 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.719444 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.718950 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.719444 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.718979 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.719444 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.719028 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.719444 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.719066 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.719444 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.719097 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.719444 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.719123 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.719444 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.719163 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.719820 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.719510 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-config-volume\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.719820 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.719553 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-config-out\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.719820 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.719582 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-web-config\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.719820 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.719626 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.719820 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.719678 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2mw5\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-kube-api-access-q2mw5\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.720045 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.719978 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.722751 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.722493 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.725064 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.725039 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-config-volume\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.725162 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.725148 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.725443 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.725417 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.725542 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.725524 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.726133 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.725674 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-config-out\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.726133 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.725820 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-web-config\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.726133 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.726089 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.726133 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.726106 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.727438 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.727419 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.727651 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.727627 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.741002 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.740646 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2mw5\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-kube-api-access-q2mw5\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:39.745394 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.744266 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f77577466-bjlt5"] Oct 31 17:27:39.748881 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:39.748848 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511ee639_1612_4e36_812e_70f7865291c1.slice/crio-b97cc610d95a14a810a8540c9c588346c8ab12383038645ca2f15bb2d9d88707 WatchSource:0}: Error finding container b97cc610d95a14a810a8540c9c588346c8ab12383038645ca2f15bb2d9d88707: Status 404 returned error can't find the container with id b97cc610d95a14a810a8540c9c588346c8ab12383038645ca2f15bb2d9d88707 Oct 31 17:27:39.912870 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:39.912821 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:40.038747 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.038715 2566 generic.go:356] "Generic (PLEG): container finished" podID="93b4b2af-0bd0-4907-9535-0ce4ef46383c" containerID="dcc90125ec09232730524eef742eb9daefc29ce81987c28639a227aaf7c5c462" exitCode=0 Oct 31 17:27:40.039094 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.038813 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66bdk" event={"ID":"93b4b2af-0bd0-4907-9535-0ce4ef46383c","Type":"ContainerDied","Data":"dcc90125ec09232730524eef742eb9daefc29ce81987c28639a227aaf7c5c462"} Oct 31 17:27:40.041295 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.041090 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x8mnz" event={"ID":"6fa107ff-c9d8-41a8-9baa-190bd8714606","Type":"ContainerStarted","Data":"12ef21ca5c241b6f759e028912fe960765929844ad16030fa15c1be500d3b4d0"} Oct 31 17:27:40.043777 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.043754 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" event={"ID":"6e895668-db69-4458-9e3d-fcd82e31084b","Type":"ContainerStarted","Data":"24f346fcf0426027d58d7d1d715c5b6513eb53a83db7bda49473c9e1baafc721"} Oct 31 17:27:40.046064 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.046041 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f77577466-bjlt5" event={"ID":"511ee639-1612-4e36-812e-70f7865291c1","Type":"ContainerStarted","Data":"b97cc610d95a14a810a8540c9c588346c8ab12383038645ca2f15bb2d9d88707"} Oct 31 17:27:40.125838 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.125795 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" podStartSLOduration=1.003406135 podStartE2EDuration="2.125781138s" totalImagesPullingTime="1.122375003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:40.12277728 +0000 UTC m=+53.019608883" watchObservedRunningTime="2026-10-31 17:27:40.125781138 +0000 UTC m=+53.022612747" Oct 31 17:27:40.162194 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.162144 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x8mnz" podStartSLOduration=2.019449711 podStartE2EDuration="5.162130858s" totalImagesPullingTime="3.142681147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:40.16150681 +0000 UTC m=+53.058338420" watchObservedRunningTime="2026-10-31 17:27:40.162130858 +0000 UTC m=+53.058962467" Oct 31 17:27:40.252864 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.252724 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 17:27:40.255870 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:40.255833 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176a27de_94c3_482d_9bcd_e335ba12400a.slice/crio-65456a2e62791904d75b78dfdd341ade59d5c954ff3eb04009ea62d80d04030b WatchSource:0}: Error finding container 65456a2e62791904d75b78dfdd341ade59d5c954ff3eb04009ea62d80d04030b: Status 404 returned error can't find the container with id 65456a2e62791904d75b78dfdd341ade59d5c954ff3eb04009ea62d80d04030b Oct 31 17:27:40.373703 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.373677 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw"] Oct 31 17:27:40.377630 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.377607 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.395074 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.394928 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 17:27:40.395074 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.394932 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 17:27:40.395074 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.395002 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 17:27:40.395074 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.395027 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 17:27:40.395074 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.395047 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dre6cbb61jri1\"" Oct 31 17:27:40.395345 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.395159 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qxdq5\"" Oct 31 17:27:40.395345 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.395173 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 17:27:40.402541 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.402461 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw"] Oct 31 17:27:40.426104 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.426075 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-tmp\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.426209 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.426126 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.426297 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.426270 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.426351 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.426320 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.426396 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.426355 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-grpc-tls\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.426396 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.426385 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-metrics-client-ca\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.426452 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.426419 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9kkw\" (UniqueName: \"kubernetes.io/projected/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-kube-api-access-b9kkw\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.426490 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.426459 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-tls\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.426677 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.426643 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.528130 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.528080 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.528383 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.528152 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-tmp\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.528383 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.528187 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.528383 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.528262 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.528383 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.528295 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.528383 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.528328 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-grpc-tls\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.528383 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.528358 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-metrics-client-ca\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.528787 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.528391 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9kkw\" (UniqueName: \"kubernetes.io/projected/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-kube-api-access-b9kkw\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.528787 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.528428 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-tls\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.530043 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.530014 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-tmp\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.530238 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.530211 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-metrics-client-ca\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.532018 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.531960 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-tls\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.532686 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.532664 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.533450 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.533429 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.533756 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.533738 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.534709 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.534689 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-grpc-tls\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.534866 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.534846 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.550877 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.550841 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9kkw\" (UniqueName: \"kubernetes.io/projected/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-kube-api-access-b9kkw\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:40.716369 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:40.716284 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:41.040265 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:41.040178 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw"] Oct 31 17:27:41.044579 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:41.044547 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016aec04_e1a2_4dd4_8a59_9cfc0418da6f.slice/crio-46895f560d2b820a7f0aa256aef97ae4677e87193f88621f3eeb2c8144b63ac5 WatchSource:0}: Error finding container 46895f560d2b820a7f0aa256aef97ae4677e87193f88621f3eeb2c8144b63ac5: Status 404 returned error can't find the container with id 46895f560d2b820a7f0aa256aef97ae4677e87193f88621f3eeb2c8144b63ac5 Oct 31 17:27:41.055739 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:41.055700 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" event={"ID":"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4","Type":"ContainerStarted","Data":"a16b2f911ecb610625a29445b8fb51a41e91c27a0ce38191b9a0d16ffd660e76"} Oct 31 17:27:41.055843 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:41.055807 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" event={"ID":"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4","Type":"ContainerStarted","Data":"a54b37a6d0a49151c00cdd59ade1ba5fd416febe487b766f258033bbf35d4eae"} Oct 31 17:27:41.055843 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:41.055836 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" event={"ID":"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4","Type":"ContainerStarted","Data":"89e0e1e4123fd899c5d223362b0836790a1d0ca8cbf1b6bf33c22bebc45f65fa"} Oct 31 17:27:41.057214 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:41.057185 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" event={"ID":"016aec04-e1a2-4dd4-8a59-9cfc0418da6f","Type":"ContainerStarted","Data":"46895f560d2b820a7f0aa256aef97ae4677e87193f88621f3eeb2c8144b63ac5"} Oct 31 17:27:41.060212 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:41.060189 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66bdk" event={"ID":"93b4b2af-0bd0-4907-9535-0ce4ef46383c","Type":"ContainerStarted","Data":"0156b54b568137b843d39af9f121891bbe34267dcec6c9593d3fda41d70558b1"} Oct 31 17:27:41.060314 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:41.060231 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66bdk" event={"ID":"93b4b2af-0bd0-4907-9535-0ce4ef46383c","Type":"ContainerStarted","Data":"5d7f343c367bae44b4bdf53ec0f5adebf67bd364abd9475e27e180457309ed73"} Oct 31 17:27:41.062269 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:41.062237 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerStarted","Data":"65456a2e62791904d75b78dfdd341ade59d5c954ff3eb04009ea62d80d04030b"} Oct 31 17:27:41.097661 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:41.097606 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" podStartSLOduration=1.666249932 podStartE2EDuration="3.097592719s" totalImagesPullingTime="1.431342787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:41.096880149 +0000 UTC m=+53.993711763" watchObservedRunningTime="2026-10-31 17:27:41.097592719 +0000 UTC m=+53.994424331" Oct 31 17:27:41.141011 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:41.140962 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-66bdk" podStartSLOduration=2.077915815 podStartE2EDuration="3.140949512s" totalImagesPullingTime="978.414518ms" totalInitContainerRuntime="84.619179ms" isStatefulPod=true podCreationTimestamp="2026-10-31 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:41.140503611 +0000 UTC m=+54.037335242" watchObservedRunningTime="2026-10-31 17:27:41.140949512 +0000 UTC m=+54.037781121" Oct 31 17:27:42.067151 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.067110 2566 generic.go:356] "Generic (PLEG): container finished" podID="176a27de-94c3-482d-9bcd-e335ba12400a" containerID="05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b" exitCode=0 Oct 31 17:27:42.069898 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.068488 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerDied","Data":"05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b"} Oct 31 17:27:42.587222 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.587190 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5db8c4796b-lwk5p"] Oct 31 17:27:42.590328 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.590298 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.610779 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.610737 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5db8c4796b-lwk5p"] Oct 31 17:27:42.622268 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.622230 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 17:27:42.622434 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.622285 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9iap3qsv6ukg3\"" Oct 31 17:27:42.622434 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.622337 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 17:27:42.622434 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.622357 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 17:27:42.623100 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.623075 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 17:27:42.623207 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.623151 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5rzbg\"" Oct 31 17:27:42.650483 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.650446 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-audit-log\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.650663 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.650495 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-metrics-server-audit-profiles\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.650663 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.650519 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-client-ca-bundle\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.650850 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.650660 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-secret-metrics-server-client-certs\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.650850 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.650698 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.650850 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.650803 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9plqg\" (UniqueName: \"kubernetes.io/projected/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-kube-api-access-9plqg\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.650966 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.650851 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-secret-metrics-server-tls\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.751730 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.751691 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-audit-log\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.751915 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.751741 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-metrics-server-audit-profiles\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.751915 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.751881 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-client-ca-bundle\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.752035 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.751958 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-secret-metrics-server-client-certs\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.752035 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.751992 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.752134 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.752112 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9plqg\" (UniqueName: \"kubernetes.io/projected/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-kube-api-access-9plqg\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.752134 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.752123 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-audit-log\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.752209 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.752165 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-secret-metrics-server-tls\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.752907 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.752879 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-metrics-server-audit-profiles\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.753036 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.752907 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.754853 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.754832 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-client-ca-bundle\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.755128 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.755110 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-secret-metrics-server-client-certs\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.756778 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.756752 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-secret-metrics-server-tls\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.773123 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.773068 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9plqg\" (UniqueName: \"kubernetes.io/projected/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-kube-api-access-9plqg\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.905759 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.905507 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:42.959192 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.959156 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v"] Oct 31 17:27:42.962836 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.962082 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:42.985387 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.984892 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 17:27:42.985387 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.985058 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rr8f2\"" Oct 31 17:27:42.986943 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:42.986878 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v"] Oct 31 17:27:43.057768 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.057736 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd2a7b5c-fbe5-43d3-9121-403946f33aca-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t2v2v\" (UID: \"bd2a7b5c-fbe5-43d3-9121-403946f33aca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:43.057937 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.057777 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmv4j\" (UniqueName: \"kubernetes.io/projected/bd2a7b5c-fbe5-43d3-9121-403946f33aca-kube-api-access-wmv4j\") pod \"monitoring-plugin-77f4f76594-t2v2v\" (UID: \"bd2a7b5c-fbe5-43d3-9121-403946f33aca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:43.074108 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.073637 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f77577466-bjlt5" event={"ID":"511ee639-1612-4e36-812e-70f7865291c1","Type":"ContainerStarted","Data":"fbc6b961e24873f71116d543cefb5b67ceede670f46ec651fa891caa3a7065fe"} Oct 31 17:27:43.158753 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.158559 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd2a7b5c-fbe5-43d3-9121-403946f33aca-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t2v2v\" (UID: \"bd2a7b5c-fbe5-43d3-9121-403946f33aca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:43.158993 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.158924 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmv4j\" (UniqueName: \"kubernetes.io/projected/bd2a7b5c-fbe5-43d3-9121-403946f33aca-kube-api-access-wmv4j\") pod \"monitoring-plugin-77f4f76594-t2v2v\" (UID: \"bd2a7b5c-fbe5-43d3-9121-403946f33aca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:43.164467 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.164437 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd2a7b5c-fbe5-43d3-9121-403946f33aca-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t2v2v\" (UID: \"bd2a7b5c-fbe5-43d3-9121-403946f33aca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:43.181372 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.181341 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmv4j\" (UniqueName: \"kubernetes.io/projected/bd2a7b5c-fbe5-43d3-9121-403946f33aca-kube-api-access-wmv4j\") pod \"monitoring-plugin-77f4f76594-t2v2v\" (UID: \"bd2a7b5c-fbe5-43d3-9121-403946f33aca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:43.226751 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.226695 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6f77577466-bjlt5" podStartSLOduration=1.180559096 podStartE2EDuration="4.226679871s" totalImagesPullingTime="3.046120775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:43.118427353 +0000 UTC m=+56.015258963" watchObservedRunningTime="2026-10-31 17:27:43.226679871 +0000 UTC m=+56.123511481" Oct 31 17:27:43.229965 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.227891 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5db8c4796b-lwk5p"] Oct 31 17:27:43.232119 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:43.232072 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a5dd3d_3ace_4a01_b358_d418d1fa81aa.slice/crio-ceb952e6b278f389afa545560d3367ef2fd9b66453f43e912bb2955cc26db04f WatchSource:0}: Error finding container ceb952e6b278f389afa545560d3367ef2fd9b66453f43e912bb2955cc26db04f: Status 404 returned error can't find the container with id ceb952e6b278f389afa545560d3367ef2fd9b66453f43e912bb2955cc26db04f Oct 31 17:27:43.274970 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.274936 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:43.413907 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.412293 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk"] Oct 31 17:27:43.416231 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.416212 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.434284 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.433208 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 17:27:43.434284 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.433431 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 17:27:43.434284 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.433508 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 17:27:43.434284 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.433525 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 17:27:43.434284 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.433564 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 17:27:43.434284 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.433675 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2xgtr\"" Oct 31 17:27:43.447852 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.447808 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 17:27:43.449536 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.449519 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk"] Oct 31 17:27:43.463330 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.463086 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.463330 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.463120 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.463330 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.463149 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-telemeter-client-tls\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.463330 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.463167 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtqcd\" (UniqueName: \"kubernetes.io/projected/2269cc9c-7732-4144-bc2e-8dafebdd1957-kube-api-access-vtqcd\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.463330 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.463197 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-serving-certs-ca-bundle\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.463330 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.463211 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-secret-telemeter-client\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.463330 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.463237 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-metrics-client-ca\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.463330 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.463310 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-federate-client-tls\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.564590 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.564557 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-telemeter-client-tls\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.564736 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.564597 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqcd\" (UniqueName: \"kubernetes.io/projected/2269cc9c-7732-4144-bc2e-8dafebdd1957-kube-api-access-vtqcd\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.564736 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.564645 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-serving-certs-ca-bundle\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.564736 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.564668 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-secret-telemeter-client\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.564736 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.564717 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-metrics-client-ca\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.564942 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.564790 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-federate-client-tls\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.564942 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.564856 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.564942 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.564888 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.566571 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.565743 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-metrics-client-ca\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.566571 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.565986 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-serving-certs-ca-bundle\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.566571 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.566523 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.567219 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.567195 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-telemeter-client-tls\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.568017 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.567992 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-federate-client-tls\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.568383 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.568362 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.569455 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.569437 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-secret-telemeter-client\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.583188 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.583158 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v"] Oct 31 17:27:43.585806 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.585781 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqcd\" (UniqueName: \"kubernetes.io/projected/2269cc9c-7732-4144-bc2e-8dafebdd1957-kube-api-access-vtqcd\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:43.587171 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:43.587148 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2a7b5c_fbe5_43d3_9121_403946f33aca.slice/crio-5682419688ec1234558e4cb49d37661d608b0ad081abb1906c9feaa7118ef0db WatchSource:0}: Error finding container 5682419688ec1234558e4cb49d37661d608b0ad081abb1906c9feaa7118ef0db: Status 404 returned error can't find the container with id 5682419688ec1234558e4cb49d37661d608b0ad081abb1906c9feaa7118ef0db Oct 31 17:27:43.741940 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:43.741915 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:44.079028 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:44.078845 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" event={"ID":"bd2a7b5c-fbe5-43d3-9121-403946f33aca","Type":"ContainerStarted","Data":"5682419688ec1234558e4cb49d37661d608b0ad081abb1906c9feaa7118ef0db"} Oct 31 17:27:44.082553 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:44.082522 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" event={"ID":"016aec04-e1a2-4dd4-8a59-9cfc0418da6f","Type":"ContainerStarted","Data":"2372afb6f297fc26497088b7172d3416d00d417ddaea8c1e2760802d870f1363"} Oct 31 17:27:44.082678 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:44.082589 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" event={"ID":"016aec04-e1a2-4dd4-8a59-9cfc0418da6f","Type":"ContainerStarted","Data":"f6e6459a15c8c11b854c9fdce52f01d8ace83ed73d3f834929283bb42336202f"} Oct 31 17:27:44.082678 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:44.082626 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" event={"ID":"016aec04-e1a2-4dd4-8a59-9cfc0418da6f","Type":"ContainerStarted","Data":"e68c9cb8fac9bfe2b2fee6f067222d211ccc7cf778ba209bb696e5bcbdbc731e"} Oct 31 17:27:44.085039 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:44.085014 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerStarted","Data":"08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8"} Oct 31 17:27:44.085144 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:44.085073 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerStarted","Data":"fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578"} Oct 31 17:27:44.085144 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:44.085098 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerStarted","Data":"5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5"} Oct 31 17:27:44.085144 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:44.085129 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerStarted","Data":"9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6"} Oct 31 17:27:44.086805 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:44.086782 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" event={"ID":"58a5dd3d-3ace-4a01-b358-d418d1fa81aa","Type":"ContainerStarted","Data":"ceb952e6b278f389afa545560d3367ef2fd9b66453f43e912bb2955cc26db04f"} Oct 31 17:27:44.090473 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:44.090451 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk"] Oct 31 17:27:44.095620 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:44.095581 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2269cc9c_7732_4144_bc2e_8dafebdd1957.slice/crio-4c6c5dcf3314c20c2a67c0d7354745a1135fbc4e1be04ef26a1e9518d2899e2b WatchSource:0}: Error finding container 4c6c5dcf3314c20c2a67c0d7354745a1135fbc4e1be04ef26a1e9518d2899e2b: Status 404 returned error can't find the container with id 4c6c5dcf3314c20c2a67c0d7354745a1135fbc4e1be04ef26a1e9518d2899e2b Oct 31 17:27:45.069360 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.069317 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 17:27:45.073549 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.073521 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.092910 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.092866 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" event={"ID":"2269cc9c-7732-4144-bc2e-8dafebdd1957","Type":"ContainerStarted","Data":"4c6c5dcf3314c20c2a67c0d7354745a1135fbc4e1be04ef26a1e9518d2899e2b"} Oct 31 17:27:45.093572 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.093541 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 17:27:45.093696 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.093635 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 17:27:45.093845 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.093829 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 17:27:45.105190 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.105163 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" event={"ID":"016aec04-e1a2-4dd4-8a59-9cfc0418da6f","Type":"ContainerStarted","Data":"8a68b76d4d776f369565468bd7d18b0d51c2084077f301df8ade74003808773e"} Oct 31 17:27:45.105336 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.105223 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" event={"ID":"016aec04-e1a2-4dd4-8a59-9cfc0418da6f","Type":"ContainerStarted","Data":"78d1343a119d893ec375c40c36700daeebd446f382c8d96cde035fd9b5ea8fa5"} Oct 31 17:27:45.105336 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.105300 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" event={"ID":"016aec04-e1a2-4dd4-8a59-9cfc0418da6f","Type":"ContainerStarted","Data":"26864b74038fcea2ebd916e70c79c2f4da534ccc5802a1a43de310079df95b54"} Oct 31 17:27:45.105556 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.105351 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:45.108115 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.108072 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 17:27:45.108115 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.108097 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 17:27:45.115776 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.112602 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerStarted","Data":"2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819"} Oct 31 17:27:45.115776 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.112769 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerStarted","Data":"50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00"} Oct 31 17:27:45.115776 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.115583 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 17:27:45.119323 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.119293 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" event={"ID":"58a5dd3d-3ace-4a01-b358-d418d1fa81aa","Type":"ContainerStarted","Data":"03c325ed45d4ca52385cb1c0f46d339fbaf89f71a8e07c7dc1c9d6bc1e4803ca"} Oct 31 17:27:45.182985 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.182913 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkkrp\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-kube-api-access-zkkrp\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.182985 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.182975 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.183182 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.183097 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-config\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.183182 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.183138 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.183182 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.183172 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.183335 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.183202 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.183335 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.183226 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.183335 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.183293 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.183676 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.183652 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.183723 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.183711 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.183763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.183737 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.184560 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.184536 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.184710 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.184686 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.184769 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.184724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.184769 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.184752 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.184870 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.184828 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.184870 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.184855 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.184940 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.184881 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.185332 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.185313 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.185528 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.185511 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 17:27:45.185528 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.185522 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ccmlejc9fapc\"" Oct 31 17:27:45.185599 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.185581 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 17:27:45.185653 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.185597 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 17:27:45.185653 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.185645 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 17:27:45.185734 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.185706 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 17:27:45.185771 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.185740 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6jpgp\"" Oct 31 17:27:45.185801 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.185781 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 17:27:45.185801 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.185787 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 17:27:45.185881 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.185818 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 17:27:45.185927 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.185898 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 17:27:45.186086 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.186067 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.189858 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.189809 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" podStartSLOduration=1.504497861 podStartE2EDuration="3.189793558s" totalImagesPullingTime="1.685295697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:45.188980021 +0000 UTC m=+58.085811630" watchObservedRunningTime="2026-10-31 17:27:45.189793558 +0000 UTC m=+58.086625170" Oct 31 17:27:45.240043 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.239971 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.060383896 podStartE2EDuration="6.239949279s" totalImagesPullingTime="3.165628415s" totalInitContainerRuntime="13.936968ms" isStatefulPod=false podCreationTimestamp="2026-10-31 17:27:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:45.238118838 +0000 UTC m=+58.134950446" watchObservedRunningTime="2026-10-31 17:27:45.239949279 +0000 UTC m=+58.136780889" Oct 31 17:27:45.287209 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287168 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.287209 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287212 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkrp\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-kube-api-access-zkkrp\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.287467 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287243 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.287467 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287307 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-config\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.287467 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287335 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.287467 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287360 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.287467 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287389 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.287467 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287424 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.287467 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287460 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.287811 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287509 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.287811 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287536 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.287811 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287572 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.287811 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.287654 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.290034 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.288479 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.290034 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.289670 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.290381 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.290430 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.290453 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.290477 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.290507 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.290531 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.290556 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.290597 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.290959 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.291379 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292071 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.291392 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292743 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.292305 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.292802 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.292761 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.293420 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.293134 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.295637 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.295500 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.304629 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.304600 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.304736 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.304676 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.307504 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.306988 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.307504 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.306997 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.307504 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.307163 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.307504 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.307418 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.307875 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.307848 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkrp\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-kube-api-access-zkkrp\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.324121 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.324082 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.324472 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.324177 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-config\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.326949 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.326924 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.403329 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.403289 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:45.737214 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.737160 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" podStartSLOduration=2.754607528 podStartE2EDuration="5.737143883s" totalImagesPullingTime="2.982536355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:27:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:45.283967327 +0000 UTC m=+58.180798936" watchObservedRunningTime="2026-10-31 17:27:45.737143883 +0000 UTC m=+58.633975492" Oct 31 17:27:45.737687 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:45.737670 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 17:27:45.739860 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:45.739829 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b556c44_090c_4e81_9d8d_2fac4026b0df.slice/crio-854c83c8123f94442c3173892727408f7e91d92df7af64c41c7148f17cb8b9cf WatchSource:0}: Error finding container 854c83c8123f94442c3173892727408f7e91d92df7af64c41c7148f17cb8b9cf: Status 404 returned error can't find the container with id 854c83c8123f94442c3173892727408f7e91d92df7af64c41c7148f17cb8b9cf Oct 31 17:27:46.124991 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:46.124949 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" event={"ID":"bd2a7b5c-fbe5-43d3-9121-403946f33aca","Type":"ContainerStarted","Data":"862e9b3895e555ea8cdaf075af6dc04d2890bc53fde45d3b3c0f5a1ae3db6d9a"} Oct 31 17:27:46.125478 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:46.125117 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:46.126725 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:46.126698 2566 generic.go:356] "Generic (PLEG): container finished" podID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerID="c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2" exitCode=0 Oct 31 17:27:46.126851 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:46.126792 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerDied","Data":"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2"} Oct 31 17:27:46.126900 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:46.126867 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerStarted","Data":"854c83c8123f94442c3173892727408f7e91d92df7af64c41c7148f17cb8b9cf"} Oct 31 17:27:46.129106 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:46.129081 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" event={"ID":"2269cc9c-7732-4144-bc2e-8dafebdd1957","Type":"ContainerStarted","Data":"8ce99ac838f911a2e68813b3220dc8af25eab109015527b2de1db0b0eb9add29"} Oct 31 17:27:46.129224 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:46.129144 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" event={"ID":"2269cc9c-7732-4144-bc2e-8dafebdd1957","Type":"ContainerStarted","Data":"8d7de3c806b58f1bddaac16ca716db9d4f2641bdb614d70c6598c05dcc4d06a7"} Oct 31 17:27:46.129224 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:46.129173 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" event={"ID":"2269cc9c-7732-4144-bc2e-8dafebdd1957","Type":"ContainerStarted","Data":"12422be59d74011107db71fc859a29829f262ac36e2e611e95bf6bb026063619"} Oct 31 17:27:46.130855 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:46.130838 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:46.165582 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:46.165530 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" podStartSLOduration=2.310545843 podStartE2EDuration="4.165516109s" totalImagesPullingTime="1.854970266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:46.164222781 +0000 UTC m=+59.061054391" watchObservedRunningTime="2026-10-31 17:27:46.165516109 +0000 UTC m=+59.062347759" Oct 31 17:27:46.259782 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:46.259685 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" podStartSLOduration=1.441815178 podStartE2EDuration="3.259671974s" totalImagesPullingTime="1.817856796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:46.258834328 +0000 UTC m=+59.155665948" watchObservedRunningTime="2026-10-31 17:27:46.259671974 +0000 UTC m=+59.156503594" Oct 31 17:27:46.808365 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:46.808332 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:48.337450 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:48.337412 2566 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f77577466-bjlt5"] Oct 31 17:27:49.049895 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:49.049867 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:49.151182 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:49.151037 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerStarted","Data":"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143"} Oct 31 17:27:49.151182 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:49.151124 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerStarted","Data":"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b"} Oct 31 17:27:49.433901 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:49.433866 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:50.161590 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:50.161551 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerStarted","Data":"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363"} Oct 31 17:27:50.161759 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:50.161619 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerStarted","Data":"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf"} Oct 31 17:27:50.161759 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:50.161645 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerStarted","Data":"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf"} Oct 31 17:27:50.161759 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:50.161668 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerStarted","Data":"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419"} Oct 31 17:27:50.231825 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:50.231761 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.403106006 podStartE2EDuration="5.23174761s" totalImagesPullingTime="2.810667249s" totalInitContainerRuntime="17.974355ms" isStatefulPod=false podCreationTimestamp="2026-10-31 17:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:50.224305659 +0000 UTC m=+63.121137268" watchObservedRunningTime="2026-10-31 17:27:50.23174761 +0000 UTC m=+63.128579245" Oct 31 17:27:50.404230 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:50.404196 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:52.150324 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.149048 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:52.462466 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.462375 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:52.462466 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.462409 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:52.477643 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.477614 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 17:27:52.477772 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.477678 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 17:27:52.485020 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.484991 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:52.485148 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.485104 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:52.563759 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.563721 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46kf\" (UniqueName: \"kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf\") pod \"network-check-target-cd9s8\" (UID: \"bcd594b6-86d2-47ff-bc81-03bd655d3954\") " pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:52.571536 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.571502 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z2hdt\"" Oct 31 17:27:52.576763 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.576742 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:52.578695 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.578675 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 17:27:52.578883 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.578865 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tj66q\"" Oct 31 17:27:52.584855 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.584833 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:52.599677 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.599650 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 17:27:52.625849 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.625816 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g46kf\" (UniqueName: \"kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf\") pod \"network-check-target-cd9s8\" (UID: \"bcd594b6-86d2-47ff-bc81-03bd655d3954\") " pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:52.874683 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.874529 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nk5tf\"" Oct 31 17:27:52.880637 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.880607 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:52.915818 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.914141 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9zfqw"] Oct 31 17:27:52.919622 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:52.919586 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30a63cf_3116_466f_8e4d_c77f6eaacb37.slice/crio-9eedba610ecc9ff1c98e484874228058cc5afa6facf80581ed305b6fe4256561 WatchSource:0}: Error finding container 9eedba610ecc9ff1c98e484874228058cc5afa6facf80581ed305b6fe4256561: Status 404 returned error can't find the container with id 9eedba610ecc9ff1c98e484874228058cc5afa6facf80581ed305b6fe4256561 Oct 31 17:27:52.923144 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:52.923122 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-swbqv"] Oct 31 17:27:53.174441 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:53.174406 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-swbqv" event={"ID":"bec5b1da-4675-4713-9e51-9e7be00b5ba0","Type":"ContainerStarted","Data":"d57583b2ea26f7df724855d27f8f2a2d7b277c8bfa6705f97b3c0c5df6294753"} Oct 31 17:27:53.176510 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:53.176474 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9zfqw" event={"ID":"d30a63cf-3116-466f-8e4d-c77f6eaacb37","Type":"ContainerStarted","Data":"9eedba610ecc9ff1c98e484874228058cc5afa6facf80581ed305b6fe4256561"} Oct 31 17:27:53.195971 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:53.195949 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cd9s8"] Oct 31 17:27:53.199099 ip-10-0-139-107 kubenswrapper[2566]: W1031 17:27:53.199068 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd594b6_86d2_47ff_bc81_03bd655d3954.slice/crio-02181171b16f1a82f937e781f5df62a0c92f2f6ac69a693da6b38a372593267c WatchSource:0}: Error finding container 02181171b16f1a82f937e781f5df62a0c92f2f6ac69a693da6b38a372593267c: Status 404 returned error can't find the container with id 02181171b16f1a82f937e781f5df62a0c92f2f6ac69a693da6b38a372593267c Oct 31 17:27:54.191083 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:54.191046 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cd9s8" event={"ID":"bcd594b6-86d2-47ff-bc81-03bd655d3954","Type":"ContainerStarted","Data":"02181171b16f1a82f937e781f5df62a0c92f2f6ac69a693da6b38a372593267c"} Oct 31 17:27:55.199540 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:55.199494 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9zfqw" event={"ID":"d30a63cf-3116-466f-8e4d-c77f6eaacb37","Type":"ContainerStarted","Data":"06bdc9a544952342d7c3730fd3fe1db17b421d997963df49d0ca72725f6d89c8"} Oct 31 17:27:55.199996 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:55.199601 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9zfqw" event={"ID":"d30a63cf-3116-466f-8e4d-c77f6eaacb37","Type":"ContainerStarted","Data":"216bd89f10cadacbbd52973bb38904a1e1f3e357143e66dfc46d81fb1fc3a670"} Oct 31 17:27:55.241853 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:55.241787 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9zfqw" podStartSLOduration=65.937775983 podStartE2EDuration="1m7.241765524s" totalImagesPullingTime="1.303989541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:55.239579869 +0000 UTC m=+68.136411479" watchObservedRunningTime="2026-10-31 17:27:55.241765524 +0000 UTC m=+68.138597134" Oct 31 17:27:57.210778 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:57.210733 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cd9s8" event={"ID":"bcd594b6-86d2-47ff-bc81-03bd655d3954","Type":"ContainerStarted","Data":"ccf10d632b05942ef840e0cb0dd9559dca86d9cbfeb43334f77a065e5a606854"} Oct 31 17:27:57.211291 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:57.210844 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:57.251598 ip-10-0-139-107 kubenswrapper[2566]: I1031 17:27:57.251541 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cd9s8" podStartSLOduration=65.733961952 podStartE2EDuration="1m9.251520435s" totalImagesPullingTime="3.517558483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:27:57.249481685 +0000 UTC m=+70.146313295" watchObservedRunningTime="2026-10-31 17:27:57.251520435 +0000 UTC m=+70.148352046" Oct 31 17:27:58.189389 ip-10-0-139-107 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 17:27:58.202647 ip-10-0-139-107 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 17:27:58.202879 ip-10-0-139-107 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 17:27:58.202940 ip-10-0-139-107 systemd[1]: kubelet.service: Consumed 4.707s CPU time. Oct 31 17:27:58.212845 ip-10-0-139-107 systemd[1]: Starting Kubernetes Kubelet... Oct 31 17:27:58.285505 ip-10-0-139-107 kubenswrapper[7618]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 17:27:58.285505 ip-10-0-139-107 kubenswrapper[7618]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 17:27:58.285505 ip-10-0-139-107 kubenswrapper[7618]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 17:27:58.285505 ip-10-0-139-107 kubenswrapper[7618]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 17:27:58.287735 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287650 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:27:58.287735 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287735 7618 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:27:58.287735 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287738 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287741 7618 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287744 7618 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287749 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287751 7618 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287754 7618 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287757 7618 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287759 7618 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287762 7618 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287765 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287768 7618 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287770 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287778 7618 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287783 7618 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287787 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287789 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287792 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287795 7618 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287798 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:27:58.287830 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287800 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287803 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287806 7618 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287809 7618 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287812 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287815 7618 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287818 7618 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287821 7618 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287823 7618 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287826 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287829 7618 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287832 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287835 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287838 7618 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287841 7618 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287844 7618 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287847 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287849 7618 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287852 7618 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:27:58.288309 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287855 7618 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287857 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287860 7618 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287862 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287865 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287868 7618 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287871 7618 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287874 7618 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287877 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287879 7618 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287882 7618 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287884 7618 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287887 7618 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287889 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287892 7618 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287894 7618 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287897 7618 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287900 7618 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287903 7618 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287906 7618 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:27:58.288766 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287909 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287911 7618 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287914 7618 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287917 7618 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287920 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287923 7618 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287926 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287928 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287931 7618 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287933 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287935 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287938 7618 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287940 7618 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287943 7618 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287946 7618 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287948 7618 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287951 7618 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287954 7618 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287956 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:27:58.289239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287959 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287963 7618 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287965 7618 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287968 7618 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287972 7618 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287976 7618 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287979 7618 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287981 7618 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287984 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287986 7618 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287990 7618 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287992 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287995 7618 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.287997 7618 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288000 7618 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288003 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288005 7618 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288008 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288010 7618 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288013 7618 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:27:58.289754 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288016 7618 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288018 7618 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288022 7618 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288024 7618 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288027 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288029 7618 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288032 7618 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288822 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288830 7618 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288833 7618 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288836 7618 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288839 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288842 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288844 7618 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288847 7618 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288849 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288852 7618 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288855 7618 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288857 7618 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288859 7618 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:27:58.290228 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288862 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288865 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288868 7618 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288870 7618 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288874 7618 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288877 7618 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288879 7618 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288882 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288884 7618 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288888 7618 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288892 7618 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288895 7618 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288899 7618 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288903 7618 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288905 7618 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288908 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288911 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288913 7618 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288916 7618 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:27:58.290748 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288919 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288921 7618 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288924 7618 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288927 7618 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288932 7618 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288936 7618 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288938 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288941 7618 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288944 7618 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288946 7618 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288948 7618 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288951 7618 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288953 7618 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288956 7618 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288959 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288961 7618 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288964 7618 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288967 7618 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288969 7618 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:27:58.291211 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288972 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288975 7618 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288977 7618 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288979 7618 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288982 7618 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288985 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288988 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288991 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288994 7618 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288997 7618 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.288999 7618 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289002 7618 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289004 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289007 7618 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289010 7618 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289013 7618 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289015 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289018 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289021 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289023 7618 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:27:58.291711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289026 7618 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289028 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289031 7618 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289033 7618 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289036 7618 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289038 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289041 7618 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289043 7618 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289046 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289048 7618 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289051 7618 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289053 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289056 7618 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289058 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289061 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289063 7618 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289066 7618 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289068 7618 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289071 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289073 7618 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:27:58.292219 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289076 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289078 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289081 7618 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289084 7618 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289086 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289089 7618 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289091 7618 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289094 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289096 7618 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289099 7618 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289101 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289104 7618 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289107 7618 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289109 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289112 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289615 7618 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289621 7618 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289624 7618 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289628 7618 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:27:58.292710 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289636 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289640 7618 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289643 7618 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289646 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289649 7618 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289651 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289654 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289661 7618 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289665 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289667 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289670 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289673 7618 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289675 7618 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289678 7618 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289682 7618 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289684 7618 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289687 7618 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289689 7618 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289692 7618 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289695 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:27:58.293204 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289698 7618 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289702 7618 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289705 7618 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289709 7618 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289712 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289716 7618 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289719 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289721 7618 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289724 7618 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289726 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289729 7618 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289731 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289735 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289740 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289742 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289745 7618 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289748 7618 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289750 7618 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289753 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289755 7618 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:27:58.293721 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289758 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289762 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289764 7618 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289767 7618 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289770 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289772 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289775 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289777 7618 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289780 7618 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289783 7618 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289785 7618 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289788 7618 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289791 7618 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289793 7618 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289796 7618 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289798 7618 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289801 7618 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289804 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289806 7618 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289809 7618 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:27:58.294238 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289812 7618 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289814 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289817 7618 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289819 7618 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289824 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289827 7618 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289830 7618 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289832 7618 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289836 7618 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289840 7618 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289844 7618 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289847 7618 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289850 7618 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289853 7618 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289856 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289858 7618 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289861 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289863 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289866 7618 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:27:58.294783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289868 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289871 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289873 7618 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289876 7618 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289878 7618 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289881 7618 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289883 7618 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289886 7618 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289889 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289892 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289895 7618 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289898 7618 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289900 7618 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289903 7618 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289905 7618 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289908 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289911 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289915 7618 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289918 7618 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:27:58.295241 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289921 7618 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289924 7618 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289927 7618 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.289929 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290018 7618 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290030 7618 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290037 7618 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290042 7618 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290046 7618 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290050 7618 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290054 7618 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290058 7618 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290061 7618 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290065 7618 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290068 7618 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290072 7618 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290075 7618 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290078 7618 flags.go:64] FLAG: --cgroup-root="" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290081 7618 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290084 7618 flags.go:64] FLAG: --client-ca-file="" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290087 7618 flags.go:64] FLAG: --cloud-provider="external" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290090 7618 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290095 7618 flags.go:64] FLAG: --cluster-domain="" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290098 7618 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 17:27:58.295708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290101 7618 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290105 7618 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290108 7618 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290112 7618 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290115 7618 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290118 7618 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290123 7618 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290127 7618 flags.go:64] FLAG: --contention-profiling="false" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290130 7618 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290133 7618 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290136 7618 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290139 7618 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290143 7618 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290147 7618 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290149 7618 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290152 7618 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290155 7618 flags.go:64] FLAG: --enable-server="true" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290159 7618 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290164 7618 flags.go:64] FLAG: --event-burst="100" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290167 7618 flags.go:64] FLAG: --event-qps="50" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290170 7618 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290174 7618 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290177 7618 flags.go:64] FLAG: --eviction-hard="" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290181 7618 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 17:27:58.296296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290184 7618 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290187 7618 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290190 7618 flags.go:64] FLAG: --eviction-soft="" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290193 7618 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290197 7618 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290200 7618 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290203 7618 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290206 7618 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290209 7618 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290212 7618 flags.go:64] FLAG: --feature-gates="" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290216 7618 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290220 7618 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290223 7618 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290227 7618 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290232 7618 flags.go:64] FLAG: --healthz-port="10248" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290235 7618 flags.go:64] FLAG: --help="false" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290238 7618 flags.go:64] FLAG: --hostname-override="ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290242 7618 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290245 7618 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290260 7618 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290264 7618 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290268 7618 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290271 7618 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 17:27:58.296875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290274 7618 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290277 7618 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290280 7618 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290283 7618 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290287 7618 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290290 7618 flags.go:64] FLAG: --kube-reserved="" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290293 7618 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290296 7618 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290300 7618 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290302 7618 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290305 7618 flags.go:64] FLAG: --lock-file="" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290308 7618 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290311 7618 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290316 7618 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290322 7618 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290325 7618 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290328 7618 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290331 7618 flags.go:64] FLAG: --logging-format="text" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290334 7618 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290338 7618 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290341 7618 flags.go:64] FLAG: --manifest-url="" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290344 7618 flags.go:64] FLAG: --manifest-url-header="" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290349 7618 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290397 7618 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290402 7618 flags.go:64] FLAG: --max-pods="110" Oct 31 17:27:58.297516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290405 7618 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290408 7618 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290411 7618 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290414 7618 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290417 7618 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290420 7618 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290423 7618 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290431 7618 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290434 7618 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290437 7618 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290440 7618 flags.go:64] FLAG: --pod-cidr="" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290443 7618 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290446 7618 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290449 7618 flags.go:64] FLAG: --pods-per-core="0" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290452 7618 flags.go:64] FLAG: --port="10250" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290455 7618 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290458 7618 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c5258cd750d2d2c5" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290461 7618 flags.go:64] FLAG: --qos-reserved="" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290465 7618 flags.go:64] FLAG: --read-only-port="10255" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290468 7618 flags.go:64] FLAG: --register-node="true" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290473 7618 flags.go:64] FLAG: --register-with-taints="" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290477 7618 flags.go:64] FLAG: --registry-burst="10" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290479 7618 flags.go:64] FLAG: --registry-qps="5" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290482 7618 flags.go:64] FLAG: --reserved-cpus="" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290485 7618 flags.go:64] FLAG: --reserved-memory="" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290489 7618 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 17:27:58.298517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290493 7618 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290496 7618 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290500 7618 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290503 7618 flags.go:64] FLAG: --runonce="false" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290508 7618 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290511 7618 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290514 7618 flags.go:64] FLAG: --seccomp-default="false" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290517 7618 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290520 7618 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290523 7618 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290526 7618 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290529 7618 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290532 7618 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290535 7618 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290538 7618 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290541 7618 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290544 7618 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290547 7618 flags.go:64] FLAG: --system-cgroups="" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290550 7618 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290556 7618 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290559 7618 flags.go:64] FLAG: --tls-cert-file="" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290562 7618 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290566 7618 flags.go:64] FLAG: --tls-min-version="" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290569 7618 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290572 7618 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 17:27:58.299198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290575 7618 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290579 7618 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290583 7618 flags.go:64] FLAG: --v="2" Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290587 7618 flags.go:64] FLAG: --version="false" Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290591 7618 flags.go:64] FLAG: --vmodule="" Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290595 7618 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.290598 7618 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290737 7618 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290741 7618 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290744 7618 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290747 7618 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290752 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290755 7618 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290758 7618 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290761 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290763 7618 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290774 7618 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290777 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290779 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290782 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:27:58.299811 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290785 7618 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290788 7618 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290791 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290794 7618 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290796 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290799 7618 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290802 7618 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290805 7618 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290807 7618 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290810 7618 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290812 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290815 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290818 7618 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290822 7618 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290824 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290827 7618 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290831 7618 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290835 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290838 7618 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:27:58.300342 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290841 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290844 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290847 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290849 7618 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290854 7618 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290857 7618 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290860 7618 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290863 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290865 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290868 7618 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290870 7618 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290873 7618 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290875 7618 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290878 7618 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290880 7618 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290883 7618 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290886 7618 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290888 7618 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290891 7618 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290893 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:27:58.300805 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290896 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290898 7618 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290901 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290904 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290907 7618 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290911 7618 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290916 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290919 7618 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290922 7618 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290925 7618 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290929 7618 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290931 7618 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290934 7618 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290937 7618 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290939 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290942 7618 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290945 7618 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290949 7618 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290951 7618 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:27:58.301290 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290954 7618 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290957 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290959 7618 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290962 7618 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290964 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290967 7618 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290969 7618 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290972 7618 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290974 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290978 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290980 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290983 7618 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290985 7618 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290988 7618 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290990 7618 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290993 7618 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290995 7618 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.290998 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291000 7618 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291004 7618 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291007 7618 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:27:58.301745 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291010 7618 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291012 7618 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291015 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291017 7618 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291020 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291022 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291025 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291027 7618 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291031 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291034 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291037 7618 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291040 7618 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291042 7618 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291045 7618 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.291073 7618 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291110 7618 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:27:58.302260 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291114 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291118 7618 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291121 7618 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291124 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291126 7618 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291129 7618 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291131 7618 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291134 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291137 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291139 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291142 7618 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291144 7618 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291147 7618 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291173 7618 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291178 7618 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291181 7618 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291183 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291186 7618 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291189 7618 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291191 7618 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291194 7618 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:27:58.302733 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291197 7618 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291200 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291203 7618 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291206 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291209 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291212 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291216 7618 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291221 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291225 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291230 7618 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291233 7618 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291235 7618 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291238 7618 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291241 7618 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291243 7618 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291246 7618 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291260 7618 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291263 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291266 7618 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:27:58.303539 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291269 7618 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291272 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291274 7618 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291277 7618 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291280 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291282 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291289 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291293 7618 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291297 7618 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291314 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291317 7618 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291320 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291323 7618 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291326 7618 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291328 7618 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291331 7618 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291334 7618 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291338 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291341 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:27:58.304083 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291344 7618 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291347 7618 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291350 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291356 7618 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291362 7618 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291369 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291374 7618 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291378 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291383 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291387 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291391 7618 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291394 7618 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291397 7618 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291400 7618 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291403 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291405 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291408 7618 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291411 7618 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291413 7618 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:27:58.304615 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291417 7618 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291420 7618 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291423 7618 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291425 7618 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291428 7618 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291430 7618 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291433 7618 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291436 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291438 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291441 7618 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291444 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291449 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291453 7618 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291458 7618 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291462 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291466 7618 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291469 7618 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291472 7618 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291474 7618 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:27:58.305231 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291477 7618 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291479 7618 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291482 7618 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291485 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291487 7618 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291490 7618 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291493 7618 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.291495 7618 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.295660 7618 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.295683 7618 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.295701 7618 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.295706 7618 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295789 7618 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295795 7618 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295799 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295802 7618 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295805 7618 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295808 7618 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295811 7618 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:27:58.305947 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295814 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295816 7618 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295819 7618 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295823 7618 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295827 7618 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295831 7618 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295834 7618 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295836 7618 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295839 7618 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295842 7618 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295845 7618 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295848 7618 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295851 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295853 7618 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295856 7618 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295859 7618 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295861 7618 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295864 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295867 7618 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:27:58.306593 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295869 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295872 7618 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295875 7618 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295885 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295888 7618 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295891 7618 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295894 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295896 7618 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295899 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295902 7618 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295904 7618 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295907 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295909 7618 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295912 7618 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295914 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295917 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295919 7618 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295922 7618 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295924 7618 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:27:58.307064 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295927 7618 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295930 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295933 7618 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295935 7618 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295940 7618 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295944 7618 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295947 7618 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295950 7618 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295953 7618 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295956 7618 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295959 7618 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295961 7618 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295964 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295967 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295970 7618 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295973 7618 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295976 7618 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295984 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295987 7618 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295990 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:27:58.307734 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295992 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295995 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.295997 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296000 7618 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296003 7618 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296006 7618 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296008 7618 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296010 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296013 7618 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296015 7618 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296018 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296020 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296023 7618 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296025 7618 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296028 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296030 7618 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296033 7618 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296036 7618 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296038 7618 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296041 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:27:58.308214 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296043 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296046 7618 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296049 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296051 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296054 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296056 7618 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296059 7618 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296061 7618 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296064 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296066 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296080 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296083 7618 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296086 7618 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296088 7618 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296091 7618 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296093 7618 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296096 7618 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296098 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296101 7618 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296103 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:27:58.308701 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296106 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.296133 7618 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296361 7618 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296368 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296371 7618 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296375 7618 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296377 7618 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296380 7618 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296383 7618 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296385 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296388 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296391 7618 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296393 7618 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296396 7618 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296398 7618 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296401 7618 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:27:58.309195 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296404 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296406 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296409 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296411 7618 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296414 7618 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296416 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296419 7618 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296427 7618 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296430 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296433 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296435 7618 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296438 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296440 7618 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296443 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296446 7618 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296448 7618 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296451 7618 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296453 7618 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296456 7618 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:27:58.309664 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296460 7618 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296464 7618 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296467 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296470 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296472 7618 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296475 7618 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296478 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296480 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296483 7618 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296485 7618 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296487 7618 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296490 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296493 7618 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296496 7618 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296498 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296501 7618 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296504 7618 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296506 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296509 7618 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296511 7618 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:27:58.310115 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296514 7618 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296522 7618 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296525 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296527 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296530 7618 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296533 7618 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296535 7618 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296538 7618 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296541 7618 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296543 7618 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296546 7618 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296548 7618 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296551 7618 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296554 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296556 7618 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296559 7618 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296562 7618 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296565 7618 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296567 7618 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296570 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:27:58.310650 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296572 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296575 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296579 7618 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296582 7618 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296585 7618 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296587 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296590 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296592 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296595 7618 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296598 7618 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296600 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296602 7618 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296605 7618 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296608 7618 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296616 7618 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296619 7618 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296621 7618 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296624 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296626 7618 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:27:58.311141 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296629 7618 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296632 7618 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296634 7618 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296637 7618 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296639 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296642 7618 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296644 7618 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296647 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296649 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296652 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296654 7618 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296657 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296660 7618 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296662 7618 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.296687 7618 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 17:27:58.311688 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296730 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296735 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296737 7618 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296740 7618 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296743 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296745 7618 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296748 7618 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296751 7618 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296753 7618 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296756 7618 feature_gate.go:386] unrecognized feature gate: Example Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296758 7618 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296761 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296764 7618 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296772 7618 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296775 7618 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296778 7618 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296780 7618 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296783 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296785 7618 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 17:27:58.312134 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296788 7618 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296791 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296793 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296796 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296798 7618 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296801 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296803 7618 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296806 7618 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296809 7618 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296812 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296814 7618 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296817 7618 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296819 7618 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296822 7618 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296824 7618 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296827 7618 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296829 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296832 7618 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296834 7618 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296840 7618 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 17:27:58.312711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296842 7618 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296845 7618 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296848 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296850 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296853 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296855 7618 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296858 7618 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296870 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296873 7618 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296876 7618 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296879 7618 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296881 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296884 7618 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296887 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296889 7618 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296892 7618 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296894 7618 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296897 7618 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296900 7618 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 17:27:58.313447 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296902 7618 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296905 7618 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296909 7618 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296913 7618 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296917 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296920 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296923 7618 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296926 7618 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296929 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296931 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296934 7618 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296936 7618 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296940 7618 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296944 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296947 7618 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296949 7618 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296952 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296954 7618 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296957 7618 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 17:27:58.314080 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296960 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296962 7618 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296970 7618 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296973 7618 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296976 7618 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296979 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296981 7618 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296984 7618 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296987 7618 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296989 7618 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296992 7618 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296994 7618 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.296997 7618 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297000 7618 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297005 7618 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297008 7618 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297011 7618 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297014 7618 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297016 7618 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297019 7618 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 17:27:58.314758 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297021 7618 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297024 7618 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297026 7618 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297029 7618 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297031 7618 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297037 7618 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297041 7618 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297044 7618 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:27:58.297047 7618 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.297112 7618 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.299572 7618 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.299663 7618 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.299860 7618 server.go:1014] "Starting client certificate rotation" Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.299996 7618 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.300059 7618 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 17:21:48 +0000 UTC" deadline="2028-06-26 11:41:38.029101391 +0000 UTC" Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.300088 7618 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14490h13m39.729015486s" Oct 31 17:27:58.315616 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.300362 7618 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 17:27:58.316389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.301572 7618 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 17:27:58.316389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.302834 7618 log.go:25] "Validated CRI v1 runtime API" Oct 31 17:27:58.316389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.305022 7618 log.go:25] "Validated CRI v1 image API" Oct 31 17:27:58.316389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.305806 7618 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 17:27:58.316389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.308336 7618 fs.go:135] Filesystem UUIDs: map[37ae1589-8032-4418-aa19-6a80e4e1c11e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b23beb05-70ea-41c7-b37b-98759e30376f:/dev/nvme0n1p4] Oct 31 17:27:58.316643 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.308359 7618 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/010c48d00c6359c0bfc73132137eda85a54cfaf74a110583fbe5d4583dc97366/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/010c48d00c6359c0bfc73132137eda85a54cfaf74a110583fbe5d4583dc97366/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02181171b16f1a82f937e781f5df62a0c92f2f6ac69a693da6b38a372593267c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02181171b16f1a82f937e781f5df62a0c92f2f6ac69a693da6b38a372593267c/userdata/shm major:0 minor:518 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/036583787fe6ee95d00c20e791023a15d74e43cd5fad9263dcb2c890f0372c01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/036583787fe6ee95d00c20e791023a15d74e43cd5fad9263dcb2c890f0372c01/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07759b19aa29bec829ba00205faa69eaa4c5b1226f243986267cda739eb23d61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07759b19aa29bec829ba00205faa69eaa4c5b1226f243986267cda739eb23d61/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f130503ff57c7c6901bc28e0fcaf54c7b3610b1fd438a136bd6f581d34752bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f130503ff57c7c6901bc28e0fcaf54c7b3610b1fd438a136bd6f581d34752bc/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f0f6f84b766e8e9c27ae5a1b15af9c9c97e80544f343a9d9a4617c66a3f383d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f0f6f84b766e8e9c27ae5a1b15af9c9c97e80544f343a9d9a4617c66a3f383d/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46895f560d2b820a7f0aa256aef97ae4677e87193f88621f3eeb2c8144b63ac5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46895f560d2b820a7f0aa256aef97ae4677e87193f88621f3eeb2c8144b63ac5/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49daaf0fda866993155a9e9a1e9d051c758c64ba9cde917ea68014e7b17ca97a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49daaf0fda866993155a9e9a1e9d051c758c64ba9cde917ea68014e7b17ca97a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c6c5dcf3314c20c2a67c0d7354745a1135fbc4e1be04ef26a1e9518d2899e2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c6c5dcf3314c20c2a67c0d7354745a1135fbc4e1be04ef26a1e9518d2899e2b/userdata/shm major:0 minor:388 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5682419688ec1234558e4cb49d37661d608b0ad081abb1906c9feaa7118ef0db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5682419688ec1234558e4cb49d37661d608b0ad081abb1906c9feaa7118ef0db/userdata/shm major:0 minor:361 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65456a2e62791904d75b78dfdd341ade59d5c954ff3eb04009ea62d80d04030b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65456a2e62791904d75b78dfdd341ade59d5c954ff3eb04009ea62d80d04030b/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6bf9846392b4c9d13b4427ecf3a61853a69972746745fbe33807869789fcc421/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6bf9846392b4c9d13b4427ecf3a61853a69972746745fbe33807869789fcc421/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a8936ba8a2ccb5380a9b3a6ceabd752ff7457d91566754d57429b10c95ced5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a8936ba8a2ccb5380a9b3a6ceabd752ff7457d91566754d57429b10c95ced5c/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c96ab1a6b8a6e25808f08deafcc74ada17b43b7b6f6a3c94b620f66b7052a79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c96ab1a6b8a6e25808f08deafcc74ada17b43b7b6f6a3c94b620f66b7052a79/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/854c83c8123f94442c3173892727408f7e91d92df7af64c41c7148f17cb8b9cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/854c83c8123f94442c3173892727408f7e91d92df7af64c41c7148f17cb8b9cf/userdata/shm major:0 minor:454 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/871100da38ff67712afa26e098f88567bdb9f934004f9067c173a20a7adcb399/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/871100da38ff67712afa26e098f88567bdb9f934004f9067c173a20a7adcb399/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94a1f08db33c8a171e666896062ed0eae9d3742fd533efb76c3c74ca5cc71816/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94a1f08db33c8a171e666896062ed0eae9d3742fd533efb76c3c74ca5cc71816/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9eedba610ecc9ff1c98e484874228058cc5afa6facf80581ed305b6fe4256561/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9eedba610ecc9ff1c98e484874228058cc5afa6facf80581ed305b6fe4256561/userdata/shm major:0 minor:513 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b97cc610d95a14a810a8540c9c588346c8ab12383038645ca2f15bb2d9d88707/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b97cc610d95a14a810a8540c9c588346c8ab12383038645ca2f15bb2d9d88707/userdata/shm major:0 minor:279 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bfccff140157cc462720964a45b0913d94475aa4ea0a5bf5d4ea226da0da30ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bfccff140157cc462720964a45b0913d94475aa4ea0a5bf5d4ea226da0da30ed/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ceb952e6b278f389afa545560d3367ef2fd9b66453f43e912bb2955cc26db04f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ceb952e6b278f389afa545560d3367ef2fd9b66453f43e912bb2955cc26db04f/userdata/shm major:0 minor:351 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d57583b2ea26f7df724855d27f8f2a2d7b277c8bfa6705f97b3c0c5df6294753/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d57583b2ea26f7df724855d27f8f2a2d7b277c8bfa6705f97b3c0c5df6294753/userdata/shm major:0 minor:515 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5989dbdb2d1c5d28a2b391e91e571a62d76150c1afe36bf4435b89aedd32b6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5989dbdb2d1c5d28a2b391e91e571a62d76150c1afe36bf4435b89aedd32b6c/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e724a097ca03b771375c87e4ac2eea048be276d8095f4b7dc5d4e14fb980c9e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e724a097ca03b771375c87e4ac2eea048be276d8095f4b7dc5d4e14fb980c9e6/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2a476a670e5c7ff9a8ec1b564638865478650ecf8fd0f9596082da8b9bf5f69/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2a476a670e5c7ff9a8ec1b564638865478650ecf8fd0f9596082da8b9bf5f69/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc3b96f14c433569b9d1c4ad0849f0e7828dba0d2d8806b7fcb408977713ce73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc3b96f14c433569b9d1c4ad0849f0e7828dba0d2d8806b7fcb408977713ce73/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/016409a4-3533-46e9-970d-4154787d4924/volumes/kubernetes.io~projected/kube-api-access-bldpx:{mountpoint:/var/lib/kubelet/pods/016409a4-3533-46e9-970d-4154787d4924/volumes/kubernetes.io~projected/kube-api-access-bldpx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~projected/kube-api-access-b9kkw:{mountpoint:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~projected/kube-api-access-b9kkw major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/159b2703-3b66-4ddc-8762-b4546b942282/volumes/kubernetes.io~projected/kube-api-access-c8d9h:{mountpoint:/var/lib/kubelet/pods/159b2703-3b66-4ddc-8762-b4546b942282/volumes/kubernetes.io~projected/kube-api-access-c8d9h major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volume-subpaths/web-config/alertmanager/9 major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volume-subpaths/web-config/config-reloader/6 major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~empty-dir/config-out major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~projected/kube-api-access-q2mw5:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~projected/kube-api-access-q2mw5 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~projected/tls-assets major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/config-volume major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/web-config major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~projected/kube-api-access-vtqcd:{mountpoint:/var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~projected/kube-api-access-vtqcd major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44b0b68e-a3e0-4bb7-b987-92b7dacd0612/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/44b0b68e-a3e0-4bb7-b987-92b7dacd0612/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44b0b68e-a3e0-4bb7-b987-92b7dacd0612/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/44b0b68e-a3e0-4bb7-b987-92b7dacd0612/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44b0b68e-a3e0-4bb7-b987-92b7dacd0612/volumes/kubernetes.io~projected/kube-api-access-blr7j:{mountpoint:/var/lib/kubelet/pods/44b0b68e-a3e0-4bb7-b987-92b7dacd0612/volumes/kubernetes.io~projected/kube-api-access-blr7j major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1/volumes/kubernetes.io~projected/kube-api-access-nrmhn:{mountpoint:/var/lib/kubelet/pods/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1/volumes/kubernetes.io~projected/kube-api-access-nrmhn major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f/volumes/kubernetes.io~projected/kube-api-access-pdrvg:{mountpoint:/var/lib/kubelet/pods/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f/volumes/kubernetes.io~projected/kube-api-access-pdrvg major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/511ee639-1612-4e36-812e-70f7865291c1/volumes/kubernetes.io~projected/kube-api-access-crxzn:{mountpoint:/var/lib/kubelet/pods/511ee639-1612-4e36-812e-70f7865291c1/volumes/kubernetes.io~projected/kube-api-access-crxzn major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/511ee639-1612-4e36-812e-70f7865291c1/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/511ee639-1612-4e36-812e-70f7865291c1/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/511ee639-1612-4e36-812e-70f7865291c1/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/511ee639-1612-4e36-812e-70f7865291c1/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58a5dd3d-3ace-4a01-b358-d418d1fa81aa/volumes/kubernetes.io~projected/kube-api-access-9plqg:{mountpoint:/var/lib/kubelet/pods/58a5dd3d-3ace-4a01-b358-d418d1fa81aa/volumes/kubernetes.io~projected/kube-api-access-9plqg major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58a5dd3d-3ace-4a01-b358-d418d1fa81aa/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/58a5dd3d-3ace-4a01-b358-d418d1fa81aa/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58a5dd3d-3ace-4a01-b358-d418d1fa81aa/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/58a5dd3d-3ace-4a01-b358-d418d1fa81aa/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58a5dd3d-3ace-4a01-b358-d418d1fa81aa/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/58a5dd3d-3ace-4a01-b358-d418d1fa81aa/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64a53072-d4ff-45d1-ab01-249f63874e30/volumes/kubernetes.io~projected/kube-api-access-txcnm:{mountpoint:/var/lib/kubelet/pods/64a53072-d4ff-45d1-ab01-249f63874e30/volumes/kubernetes.io~projected/kube-api-access-txcnm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e895668-db69-4458-9e3d-fcd82e31084b/volumes/kubernetes.io~projected/kube-api-access-8nkrt:{mountpoint:/var/lib/kubelet/pods/6e895668-db69-4458-9e3d-fcd82e31084b/volumes/kubernetes.io~projected/kube-api-access-8nkrt major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e895668-db69-4458-9e3d-fcd82e31084b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6e895668-db69-4458-9e3d-fcd82e31084b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e895668-db69-4458-9e3d-fcd82e31084b/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/6e895668-db69-4458-9e3d-fcd82e31084b/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fa107ff-c9d8-41a8-9baa-190bd8714606/volumes/kubernetes.io~projected/kube-api-access-rdphq:{mountpoint:/var/lib/kubelet/pods/6fa107ff-c9d8-41a8-9baa-190bd8714606/volumes/kubernetes.io~projected/kube-api-access-rdphq major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fa107ff-c9d8-41a8-9baa-190bd8714606/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6fa107ff-c9d8-41a8-9baa-190bd8714606/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76940cd0-ef63-4e10-965e-e871ab476281/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/76940cd0-ef63-4e10-965e-e871ab476281/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/795fdffb-4631-44b0-a0fa-91b74548f1e9/volumes/kubernetes.io~projected/kube-api-access-w9tw2:{mountpoint:/var/lib/kubelet/pods/795fdffb-4631-44b0-a0fa-91b74548f1e9/volumes/kubernetes.io~projected/kube-api-access-w9tw2 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/795fdffb-4631-44b0-a0fa-91b74548f1e9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/795fdffb-4631-44b0-a0fa-91b74548f1e9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a3fc242-f912-4cdd-8f92-dcd8319ba40a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7a3fc242-f912-4cdd-8f92-dcd8319ba40a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a3fc242-f912-4cdd-8f92-dcd8319ba40a/volumes/kubernetes.io~projected/kube-api-access-qq4bl:{mountpoint:/var/lib/kubelet/pods/7a3fc242-f912-4cdd-8f92-dcd8319ba40a/volumes/kubernetes.io~projected/kube-api-access-qq4bl major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a3fc242-f912-4cdd-8f92-dcd8319ba40a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7a3fc242-f912-4cdd-8f92-dcd8319ba40a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/806f6a75-275f-48d2-ae95-03e1a5e46515/volumes/kubernetes.io~projected/kube-api-access-wwxk9:{mountpoint:/var/lib/kubelet/pods/806f6a75-275f-48d2-ae95-03e1a5e46515/volumes/kubernetes.io~projected/kube-api-access-wwxk9 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4/volumes/kubernetes.io~projected/kube-api-access-jx2wp:{mountpoint:/var/lib/kubelet/pods/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4/volumes/kubernetes.io~projected/kube-api-access-jx2wp major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93b4b2af-0bd0-4907-9535-0ce4ef46383c/volumes/kubernetes.io~projected/kube-api-access-hh6wh:{mountpoint:/var/lib/kubelet/pods/93b4b2af-0bd0-4907-9535-0ce4ef46383c/volumes/kubernetes.io~projected/kube-api-access-hh6wh major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93b4b2af-0bd0-4907-9535-0ce4ef46383c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/93b4b2af-0bd0-4907-9535-0ce4ef46383c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93b4b2af-0bd0-4907-9535-0ce4ef46383c/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/93b4b2af-0bd0-4907-9535-0ce4ef46383c/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volume-subpaths/web-config/config-reloader/2 major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volume-subpaths/web-config/prometheus/15 major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~empty-dir/config-out major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~projected/kube-api-access-zkkrp:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~projected/kube-api-access-zkkrp major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~projected/tls-assets major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/config major:0 minor:448 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:451 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:447 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/web-config major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcd594b6-86d2-47ff-bc81-03bd655d3954/volumes/kubernetes.io~projected/kube-api-access-g46kf:{mountpoint:/var/lib/kubelet/pods/bcd594b6-86d2-47ff-bc81-03bd655d3954/volumes/kubernetes.io~projected/kube-api-access-g46kf major:0 minor:517 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0/volumes/kubernetes.io~projected/kube-api-access-x7s8n:{mountpoint:/var/lib/kubelet/pods/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0/volumes/kubernetes.io~projected/kube-api-access-x7s8n major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd2a7b5c-fbe5-43d3-9121-403946f33aca/volumes/kubernetes.io~projected/kube-api-access-wmv4j:{mountpoint:/var/lib/kubelet/pods/bd2a7b5c-fbe5-43d3-9121-403946f33aca/volumes/kubernetes.io~projected/kube-api-access-wmv4j major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd2a7b5c-fbe5-43d3-9121-403946f33aca/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/bd2a7b5c-fbe5-43d3-9121-403946f33aca/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bec5b1da-4675-4713-9e51-9e7be00b5ba0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bec5b1da-4675-4713-9e51-9e7be00b5ba0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:508 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d30a63cf-3116-466f-8e4d-c77f6eaacb37/volumes/kubernetes.io~projected/kube-api-access-n4x98:{mountpoint:/var/lib/kubelet/pods/d30a63cf-3116-466f-8e4d-c77f6eaacb37/volumes/kubernetes.io~projected/kube-api-access-n4x98 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d30a63cf-3116-466f-8e4d-c77f6eaacb37/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/d30a63cf-3116-466f-8e4d-c77f6eaacb37/volumes/kubernetes.io~secret/metrics-certs major:0 minor:512 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/bb9b2ebbcf0e75f238662b9afe3c4e74bccad51aa83e885309b602f0888736c0/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/38561c7aa11cd8c621f2b31c70815e1a3b23f402673c9659d77f54def85deb58/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d74447ed31b030a1d7c2b4a18ee8d39b811dd5364309b10244c24d64c0892fcb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/65d75ff6564af065f3c340192c0dcd253dcf3ea7dd01b18e3e54d185ef9c5e7d/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5ba3c2e67270f3c00310c76c749d05eb25d77fbaebd68c14fb9f870a646a2cb9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1cf86d93133cb8b4a8d89dffd55637bd788bed78b2afd37df89f551b6b016288/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/704735ab55faf09b54c16680f2fca47c37b5927491161338f4d78da2fc7d4bf4/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d8f0cee373f9e5b10788ce97410ac255147e6696301948cef0617587ef3a05a7/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/822714c7018245e71c279ae62f7309b4c6912a01dda6c7e6ca537c916e96598b/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/a3d48a413b1b3d77456fa513b3591dd7dc26f01888b051114573dcb688a08de2/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/66bbc07d24e1f39d1d3b5f063885d7b3b804e4bbf3ca95dd93ff2e7a9fd8b348/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/8efde3539df80fd1cfa4258eea3c5989c9ba21468c5194344acbc36c231e34b4/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/7dcd7a2425a1355fcf5d0a2ce729aecd12344785797eba3a2c41c13de8866102/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1721b08ae97c734028a13f9abc5f3884d6f15c660bcdea543924aa40bcb34f8f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9fd6d5558a8c9c8879ed74b27729b3d431c8f143176a8cae9d91a0703a04cbfa/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1998fc519b9dec3775a59b285eb2727d2a5e8aed3ba8adfa7f64cb941600e5be/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c823bbbf5e9f2b11bbb66e91f72ab81f0fb702dbbc8744d2464d999cedf59c94/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e01d08455002351edb6a125e45f21e24dc764fd9da4a1f4f5c33556743ad7e09/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/69863eb78e4c692244afc05c43a04e1eec4a95cca2b9a04f57f64a55c82d7274/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5a593a680b85e9652d3d791ae048f23d7f63f3e939d70e0e0b771007fd5f3443/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/31227873828d11bbc90bba9a8c512ade41c9f8475823aeb150c45aada65614de/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/650a0c078371433447f349a5f6c3f3f4ce2a7a9c14fd9d7686d83ac38034dd6f/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/8cd166b9b3ae9c5fa1f938c79bed6d226db966da226b6f20fdba61b34e60f6f6/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/225ff6ee878284971bea4749d9967cf01e7f8160e2638b3bc7250ed2815e4a44/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/9b31c18e75b3fd2dd2813d4de60adbd3c5cc174e8593c825c0c8147b42281f62/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/a2ec75113d660692342e65c9167c03c3849c1ac7ffac0216eef0da5d77464f88/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/763f252d87d545c1df96b6cd1d247a744dacf443ccb3c93929a102bebc086bad/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/5033e46f6ef81b37ceddaeda1f9f3a16e47818c1d06184c98a6ba3c63e533a0a/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/2597bd2eae2070e3555c7f6683ba92d198948c188c9cbd5a4bd10d0797e936be/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/e7187f9d2acf5fc40da797e870811847f946bf361e02660c97294d1177bba05c/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/8bbf5c0a4f55ba37f28128acf84383ca00dd39239597b302df5add45f6c17d30/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/9c2b90ca577dfff15966ce17fe97c3564ff6e355d1ff8a144d53503ce44c5eef/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/fdcbddb43cf31362d767f074a43951be3fed382ae0679ffa637e65b3f1bd3198/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/bd56547c038f7d8ed5275c1b416561365b9958b53a1b342be0fe6b6b744c5014/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/ac816e0367052ecc4e728768629dfafd9302642f46f18aae46830ba6b74c4baf/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/70f8e5765c561dbcae176be1b38d700072104e6cf54691f27b71c59cb6a925d7/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/91b84bb7d8a4b1698314305d53db836d5063686bf126ed8b5a5b60fa42bf3654/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/4e270e572b582fc61efaa23d188345ba8da822a1c25269a8a745e51f5ed4eaa3/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/511132a7eb1743aba0cc7ef46bd1d50d3c559d5d8fdb5151a66856959414f983/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/e394f48ac7c5f85c968546c552c6fb72e8b11726eaf9e24dc8903c7c205aad91/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/36467e4391dcb8ea7f0f1713957b3081843f33b6c9d13c29028bd403f1789e77/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/d2868326f4c16f53d75090724eb45a8d8c2a496c1cdefde0267e873d1a93a215/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/debf9fb6acc764d4d285d9c81cd8deb3ea7e5842e30346f1c1e4128ab7ea5c41/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/fccff737bc96919a762e483d068d398f5862a01fabb04a7861858e8b2da4fe92/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/87491ebed787b0c94c456e00bcc4011cf1b33c3deb80e7f1da5fa5bfe8224a80/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/fb9ea61eeb6443e745cc4fcf99404d77c4a07b1184d9b792e924c51e5898b923/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/32e63024b0669deed9b76223c17c08b98b1fb1f924867d6438e10e1d406de7ce/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/776679eda493ccb2742a2b61c14842c98e944248212b24edb1f3b0caf0ba0dac/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/678e3e504bc5a8e25a9800817e542369043d28ff9ddc2cfafa3b71fcdfa453bd/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/1fb3bf2391bc004f64a84efb736794b7752048917fa8dfd4947c9a13bdbb89b2/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/5895d967843b2d1ba93b59ff6e5d50e9892ed9a70a0fd384ef993ee227614787/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/147c995775c789962164be9cbecba4b71c008017c98c87a3fd5c51fe6de4e403/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/dcd30fafd7f21b2d0df6449ead150d693c26c85aa858fd813427e746a76dbda6/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/ad85b9698ceb23af9adc2dd70e0332dd317e554033139c91ced7143d0a2f4b9f/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/d2c5849057a50a76e9042b17bad0718ded9d6ddf733ca888de0ba42877d5803f/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/c91666885da18f43ef1bb5dc77ad9fe4beeaabf34841f28d4c1902d87ba54b36/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/52aa63f80759f930373e1b5e27dd6e19a95cbf23dae8f3482d2021cf32dd51c9/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/fd3d355e8155d39a559702a797369c40cb382d810c41b8c17cf5b73117435861/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/8f8e474669a1c191b896de2944959f1235eafc402514a4f861faa023bd368367/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/acec3ff5d283d0d7f91a50912386ed26747a88a07fbc3c60a5582653295f5d83/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/3fcd6d23d6e689cead221e3c1544b69f98fb8d53a9d2ec7dd8748c96c1061932/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/82de5602f1d30f708c61201acf0858370af4e3e434a7448d848e9413aa52496b/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-458:{mountpoint:/var/lib/containers/storage/overlay/575df7c1b1f464e9303f185fcef0b71ce22896e0f715cc5b2a6e1ba39f313690/merged major:0 minor:458 fsType:overlay blockSize:0} overlay_0-464:{mountpoint:/var/lib/containers/storage/overlay/f02e90a16dded5aa200edd7bfa671c9793b8342c86ffc9f228ee80b06d5abef1/merged major:0 minor:464 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/854917985e60bfcdb41bebba3a4257e8058909b8277ad07ae712cebc1cf65427/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-472:{mountpoint:/var/lib/containers/storage/overlay/78dce443d8bc38e8be56650fc0c0146005d1248199be0b7f47cb925beb1cc7d3/merged major:0 minor:472 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/f97ec360f0d1ab05420b2686ca43c74d566e455d2285e257059514deb048cc85/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/e005e4f1d94787959f88df4d845582ef0d52352aa320ac01a277edc032a925a8/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/174edae911f6656b134708e2234b52dbe9947f880ebd0c299b30ef523314716b/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/49ad61738821085109d020607baf87a296425829e76c1743baadf8a7854e48b0/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-498:{mountpoint:/var/lib/containers/storage/overlay/77739e040c69542ebc55d6597d1d53e4440d174ff63591dec1d760a9a3cbe59c/merged major:0 minor:498 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/7a0eb6f46f0396503501574a1ddc265241be5c1640d67904aa576fd7a5b8024f/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/bc0b7c284263883e79f865e1b634adbb4217e13af06d6947cde7baed48951c8d/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-520:{mountpoint:/var/lib/containers/storage/overlay/b366838b2d7badf4f343673a0dd5f9350b72281d86677ead44c9417097538df6/merged major:0 minor:520 fsType:overlay blockSize:0} overlay_0-522:{mountpoint:/var/lib/containers/storage/overlay/f6226bfa3913bce159b76727bca1646a8efd438a75ae6cc83fc2c7931727c3e7/merged major:0 minor:522 fsType:overlay blockSize:0} overlay_0-524:{mountpoint:/var/lib/containers/storage/overlay/6a97d8d12abd991ca8adbb52cec2c5cf31bd773b5d2d278f382d0f71302f12bc/merged major:0 minor:524 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/42013046b640808d57fa68983a26ded36cd605a33d92fa3c030fd878c084532b/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-528:{mountpoint:/var/lib/containers/storage/overlay/73fe45200ceb31a81345af8dda0efb631c2d7cc83f76d7a4e5b01fbd71673622/merged major:0 minor:528 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/bbf7b69f50fe0ef8f6c15796d78b62f4cb2407452da9d2aeafbf8aa1f8ee5d55/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-534:{mountpoint:/var/lib/containers/storage/overlay/c0368cfdde1116bde25ce47ac48920f9f2bf631a00da7d7fdf0c9f642f097a77/merged major:0 minor:534 fsType:overlay blockSize:0} overlay_0-539:{mountpoint:/var/lib/containers/storage/overlay/bc52ef2c142c7df361c242e65f11464971d55c7aafdeecdc2ca9a3d2f6c2d158/merged major:0 minor:539 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1a0d3fcb2d683ac4f823770911cfa111e6e94a37c4c2be812bfd60eb070dd68c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/08cf4c3933614409b0674527ad6d85c8f532d816ec1b934c321416d6362be208/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/1b8650642bfee5112d7d973b4c68317eea55343d158b6c2a9ad4627d4e1bb3c1/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/61f198609e7cb09b1940ac9bc182b4d6f601bd41ab3ad39e93fb2cbe08627a1b/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/0e2833ad54f4e07db8278035cb864efc38f4f0834afdc0701d430d0f3c3de75e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0d020896b3194a8eb1a7277a3c9081c455aefefd669ffc78bdc730c9c14766f6/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/eb2474d4225a1f9bf0162ae721d000074cc1feafdc6a199107dc7f231558dbf4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9a374ea0dc100e330f1c6dc861191aefc1baca53eca089447b81179f9dfe047e/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 17:27:58.317997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.317519 7618 manager.go:217] Machine: {Timestamp:2026-10-31 17:27:58.316998809 +0000 UTC m=+0.064973266 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099828 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2193e5e020c7c03bc4b733ed2f13b4 SystemUUID:ec2193e5-e020-c7c0-3bc4-b733ed2f13b4 BootID:33afd093-d3e1-4d3b-ba37-70e9e8cc1013 Filesystems:[{Device:/var/lib/kubelet/pods/64a53072-d4ff-45d1-ab01-249f63874e30/volumes/kubernetes.io~projected/kube-api-access-txcnm DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a8936ba8a2ccb5380a9b3a6ceabd752ff7457d91566754d57429b10c95ced5c/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/511ee639-1612-4e36-812e-70f7865291c1/volumes/kubernetes.io~projected/kube-api-access-crxzn DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-524 DeviceMajor:0 DeviceMinor:524 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-539 DeviceMajor:0 DeviceMinor:539 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f130503ff57c7c6901bc28e0fcaf54c7b3610b1fd438a136bd6f581d34752bc/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e895668-db69-4458-9e3d-fcd82e31084b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e724a097ca03b771375c87e4ac2eea048be276d8095f4b7dc5d4e14fb980c9e6/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:453 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44b0b68e-a3e0-4bb7-b987-92b7dacd0612/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93b4b2af-0bd0-4907-9535-0ce4ef46383c/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ceb952e6b278f389afa545560d3367ef2fd9b66453f43e912bb2955cc26db04f/userdata/shm DeviceMajor:0 DeviceMinor:351 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:438 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:453 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bec5b1da-4675-4713-9e51-9e7be00b5ba0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:508 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/76940cd0-ef63-4e10-965e-e871ab476281/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46895f560d2b820a7f0aa256aef97ae4677e87193f88621f3eeb2c8144b63ac5/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1/volumes/kubernetes.io~projected/kube-api-access-nrmhn DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c96ab1a6b8a6e25808f08deafcc74ada17b43b7b6f6a3c94b620f66b7052a79/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:445 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d30a63cf-3116-466f-8e4d-c77f6eaacb37/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:512 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d57583b2ea26f7df724855d27f8f2a2d7b277c8bfa6705f97b3c0c5df6294753/userdata/shm DeviceMajor:0 DeviceMinor:515 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4/volumes/kubernetes.io~projected/kube-api-access-jx2wp DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b97cc610d95a14a810a8540c9c588346c8ab12383038645ca2f15bb2d9d88707/userdata/shm DeviceMajor:0 DeviceMinor:279 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58a5dd3d-3ace-4a01-b358-d418d1fa81aa/volumes/kubernetes.io~projected/kube-api-access-9plqg DeviceMajor:0 DeviceMinor:348 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:444 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:447 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58a5dd3d-3ace-4a01-b358-d418d1fa81aa/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:346 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~projected/kube-api-access-zkkrp DeviceMajor:0 DeviceMinor:452 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02181171b16f1a82f937e781f5df62a0c92f2f6ac69a693da6b38a372593267c/userdata/shm DeviceMajor:0 DeviceMinor:518 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f0f6f84b766e8e9c27ae5a1b15af9c9c97e80544f343a9d9a4617c66a3f383d/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/036583787fe6ee95d00c20e791023a15d74e43cd5fad9263dcb2c890f0372c01/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6bf9846392b4c9d13b4427ecf3a61853a69972746745fbe33807869789fcc421/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5682419688ec1234558e4cb49d37661d608b0ad081abb1906c9feaa7118ef0db/userdata/shm DeviceMajor:0 DeviceMinor:361 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:376 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~projected/kube-api-access-vtqcd DeviceMajor:0 DeviceMinor:383 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:446 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5989dbdb2d1c5d28a2b391e91e571a62d76150c1afe36bf4435b89aedd32b6c/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:380 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f/volumes/kubernetes.io~projected/kube-api-access-pdrvg DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/806f6a75-275f-48d2-ae95-03e1a5e46515/volumes/kubernetes.io~projected/kube-api-access-wwxk9 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bfccff140157cc462720964a45b0913d94475aa4ea0a5bf5d4ea226da0da30ed/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/871100da38ff67712afa26e098f88567bdb9f934004f9067c173a20a7adcb399/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/795fdffb-4631-44b0-a0fa-91b74548f1e9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-534 DeviceMajor:0 DeviceMinor:534 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/016409a4-3533-46e9-970d-4154787d4924/volumes/kubernetes.io~projected/kube-api-access-bldpx DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2a476a670e5c7ff9a8ec1b564638865478650ecf8fd0f9596082da8b9bf5f69/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/795fdffb-4631-44b0-a0fa-91b74548f1e9/volumes/kubernetes.io~projected/kube-api-access-w9tw2 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-528 DeviceMajor:0 DeviceMinor:528 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6fa107ff-c9d8-41a8-9baa-190bd8714606/volumes/kubernetes.io~projected/kube-api-access-rdphq DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58a5dd3d-3ace-4a01-b358-d418d1fa81aa/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:345 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc3b96f14c433569b9d1c4ad0849f0e7828dba0d2d8806b7fcb408977713ce73/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7a3fc242-f912-4cdd-8f92-dcd8319ba40a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/93b4b2af-0bd0-4907-9535-0ce4ef46383c/volumes/kubernetes.io~projected/kube-api-access-hh6wh DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-464 DeviceMajor:0 DeviceMinor:464 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:453 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-522 DeviceMajor:0 DeviceMinor:522 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6fa107ff-c9d8-41a8-9baa-190bd8714606/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:292 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94a1f08db33c8a171e666896062ed0eae9d3742fd533efb76c3c74ca5cc71816/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~projected/kube-api-access-q2mw5 DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-472 DeviceMajor:0 DeviceMinor:472 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49daaf0fda866993155a9e9a1e9d051c758c64ba9cde917ea68014e7b17ca97a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/44b0b68e-a3e0-4bb7-b987-92b7dacd0612/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93b4b2af-0bd0-4907-9535-0ce4ef46383c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:382 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e895668-db69-4458-9e3d-fcd82e31084b/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/511ee639-1612-4e36-812e-70f7865291c1/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:273 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-520 DeviceMajor:0 DeviceMinor:520 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07759b19aa29bec829ba00205faa69eaa4c5b1226f243986267cda739eb23d61/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd2a7b5c-fbe5-43d3-9121-403946f33aca/volumes/kubernetes.io~projected/kube-api-access-wmv4j DeviceMajor:0 DeviceMinor:358 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:450 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58a5dd3d-3ace-4a01-b358-d418d1fa81aa/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:347 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:443 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/159b2703-3b66-4ddc-8762-b4546b942282/volumes/kubernetes.io~projected/kube-api-access-c8d9h DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0/volumes/kubernetes.io~projected/kube-api-access-x7s8n DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:451 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:449 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-458 DeviceMajor:0 DeviceMinor:458 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/44b0b68e-a3e0-4bb7-b987-92b7dacd0612/volumes/kubernetes.io~projected/kube-api-access-blr7j DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e895668-db69-4458-9e3d-fcd82e31084b/volumes/kubernetes.io~projected/kube-api-access-8nkrt DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~projected/kube-api-access-b9kkw DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:448 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/854c83c8123f94442c3173892727408f7e91d92df7af64c41c7148f17cb8b9cf/userdata/shm DeviceMajor:0 DeviceMinor:454 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-498 DeviceMajor:0 DeviceMinor:498 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d30a63cf-3116-466f-8e4d-c77f6eaacb37/volumes/kubernetes.io~projected/kube-api-access-n4x98 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9eedba610ecc9ff1c98e484874228058cc5afa6facf80581ed305b6fe4256561/userdata/shm DeviceMajor:0 DeviceMinor:513 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65456a2e62791904d75b78dfdd341ade59d5c954ff3eb04009ea62d80d04030b/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/016aec04-e1a2-4dd4-8a59-9cfc0418da6f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd2a7b5c-fbe5-43d3-9121-403946f33aca/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:357 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2269cc9c-7732-4144-bc2e-8dafebdd1957/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:381 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7a3fc242-f912-4cdd-8f92-dcd8319ba40a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7a3fc242-f912-4cdd-8f92-dcd8319ba40a/volumes/kubernetes.io~projected/kube-api-access-qq4bl DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/511ee639-1612-4e36-812e-70f7865291c1/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:277 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c6c5dcf3314c20c2a67c0d7354745a1135fbc4e1be04ef26a1e9518d2899e2b/userdata/shm DeviceMajor:0 DeviceMinor:388 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:453 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/010c48d00c6359c0bfc73132137eda85a54cfaf74a110583fbe5d4583dc97366/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bcd594b6-86d2-47ff-bc81-03bd655d3954/volumes/kubernetes.io~projected/kube-api-access-g46kf DeviceMajor:0 DeviceMinor:517 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02181171b16f1a8 MacAddress:4a:3b:3a:9d:ec:6d Speed:10000 Mtu:8901} {Name:07759b19aa29bec MacAddress:0a:3b:78:31:32:40 Speed:10000 Mtu:8901} {Name:46895f560d2b820 MacAddress:2e:e0:5d:6d:31:56 Speed:10000 Mtu:8901} {Name:4c6c5dcf3314c20 MacAddress:02:a1:9c:0b:34:48 Speed:10000 Mtu:8901} {Name:5682419688ec123 MacAddress:f6:b6:70:5f:6b:e5 Speed:10000 Mtu:8901} {Name:65456a2e6279190 MacAddress:6a:ae:f0:58:25:c8 Speed:10000 Mtu:8901} {Name:6bf9846392b4c9d MacAddress:56:c7:4d:1d:12:7b Speed:10000 Mtu:8901} {Name:7c96ab1a6b8a6e2 MacAddress:3e:47:93:3a:2e:97 Speed:10000 Mtu:8901} {Name:854c83c8123f944 MacAddress:e6:3e:01:95:b8:ad Speed:10000 Mtu:8901} {Name:871100da38ff677 MacAddress:fa:17:e9:7d:29:af Speed:10000 Mtu:8901} {Name:9eedba610ecc9ff MacAddress:16:26:fc:fc:41:5f Speed:10000 Mtu:8901} {Name:b97cc610d95a14a MacAddress:72:67:9b:42:81:be Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c1:f9:ad:15 Speed:0 Mtu:9001} {Name:br-int MacAddress:42:62:ef:3f:f3:79 Speed:0 Mtu:8901} {Name:ceb952e6b278f38 MacAddress:e2:aa:cd:71:cb:51 Speed:10000 Mtu:8901} {Name:d57583b2ea26f7d MacAddress:8e:29:88:8e:df:62 Speed:10000 Mtu:8901} {Name:e724a097ca03b77 MacAddress:fe:d3:0c:b9:61:1d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c1:f9:ad:15 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:1e:03:c6:10:7a:95 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:da:d5:44:94:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 17:27:58.318300 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318003 7618 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 17:27:58.318300 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318055 7618 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 17:27:58.318361 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318343 7618 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 17:27:58.318495 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318365 7618 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-107.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 17:27:58.318537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318497 7618 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 17:27:58.318537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318505 7618 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 17:27:58.318537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318520 7618 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 17:27:58.318537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318532 7618 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 17:27:58.318639 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318617 7618 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 17:27:58.318781 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318763 7618 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 17:27:58.318879 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318870 7618 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 17:27:58.318914 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318905 7618 kubelet.go:493] "Attempting to sync node with API server" Oct 31 17:27:58.318992 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318918 7618 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 17:27:58.318992 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318928 7618 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 17:27:58.318992 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318936 7618 kubelet.go:405] "Adding apiserver pod source" Oct 31 17:27:58.318992 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.318945 7618 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 17:27:58.319810 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.319794 7618 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 17:27:58.319810 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.319810 7618 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 17:27:58.320239 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.320226 7618 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 17:27:58.323006 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.322992 7618 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 17:27:58.323006 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323005 7618 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 17:27:58.323108 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323099 7618 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 17:27:58.323307 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323297 7618 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 17:27:58.323339 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323326 7618 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 17:27:58.323426 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323417 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 17:27:58.323461 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323431 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 17:27:58.323461 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323438 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 17:27:58.323461 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323443 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 17:27:58.323461 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323450 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 17:27:58.323461 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323456 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 17:27:58.323592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323474 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 17:27:58.323592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323484 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 17:27:58.323592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323491 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 17:27:58.323592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323497 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 17:27:58.323592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323513 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 17:27:58.323592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323523 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 17:27:58.323592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323548 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 17:27:58.323592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.323557 7618 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 17:27:58.327058 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.327043 7618 server.go:1263] "Started kubelet" Oct 31 17:27:58.327170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.327110 7618 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 17:27:58.327223 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.327142 7618 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 17:27:58.327284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.327248 7618 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 17:27:58.327642 ip-10-0-139-107 systemd[1]: Started Kubernetes Kubelet. Oct 31 17:27:58.328648 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.328631 7618 server.go:317] "Adding debug handlers to kubelet server" Oct 31 17:27:58.328972 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.328945 7618 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 17:27:58.333162 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.333139 7618 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 17:27:58.333288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.333193 7618 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 17:27:58.333288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.333222 7618 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 17:27:58.333288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.333238 7618 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 17:27:58.333476 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.333340 7618 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 17:27:58.333476 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.333356 7618 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 17:21:49 +0000 UTC" deadline="2028-04-01 05:54:34.969796337 +0000 UTC" Oct 31 17:27:58.333476 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.333376 7618 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12420h26m36.636422762s" Oct 31 17:27:58.334658 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:58.334171 7618 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Oct 31 17:27:58.335069 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.334860 7618 factory.go:55] Registering systemd factory Oct 31 17:27:58.335069 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.334884 7618 factory.go:223] Registration of the systemd container factory successfully Oct 31 17:27:58.337393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.335361 7618 factory.go:153] Registering CRI-O factory Oct 31 17:27:58.337393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.335379 7618 factory.go:223] Registration of the crio container factory successfully Oct 31 17:27:58.337393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.335433 7618 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 17:27:58.337393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.335455 7618 factory.go:103] Registering Raw factory Oct 31 17:27:58.337393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.335470 7618 manager.go:1199] Started watching for new ooms in manager Oct 31 17:27:58.337393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.335791 7618 manager.go:319] Starting recovery of all containers Oct 31 17:27:58.337884 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.337844 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44b0b68e-a3e0-4bb7-b987-92b7dacd0612" volumeName="kubernetes.io/empty-dir/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-tuned" seLinuxMountContext="" Oct 31 17:27:58.337946 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.337888 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" volumeName="kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 17:27:58.337946 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.337905 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 17:27:58.337946 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.337918 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-config" seLinuxMountContext="" Oct 31 17:27:58.337946 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.337931 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e895668-db69-4458-9e3d-fcd82e31084b" volumeName="kubernetes.io/projected/6e895668-db69-4458-9e3d-fcd82e31084b-kube-api-access-8nkrt" seLinuxMountContext="" Oct 31 17:27:58.338082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.337945 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" volumeName="kubernetes.io/projected/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-kube-api-access-pdrvg" seLinuxMountContext="" Oct 31 17:27:58.338082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.337958 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="511ee639-1612-4e36-812e-70f7865291c1" volumeName="kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle" seLinuxMountContext="" Oct 31 17:27:58.338082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.337971 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="511ee639-1612-4e36-812e-70f7865291c1" volumeName="kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config" seLinuxMountContext="" Oct 31 17:27:58.338082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.337987 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e895668-db69-4458-9e3d-fcd82e31084b" volumeName="kubernetes.io/configmap/6e895668-db69-4458-9e3d-fcd82e31084b-metrics-client-ca" seLinuxMountContext="" Oct 31 17:27:58.338082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.337999 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a3fc242-f912-4cdd-8f92-dcd8319ba40a" volumeName="kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-env-overrides" seLinuxMountContext="" Oct 31 17:27:58.338082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338011 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd594b6-86d2-47ff-bc81-03bd655d3954" volumeName="kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf" seLinuxMountContext="" Oct 31 17:27:58.338082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338023 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b4b2af-0bd0-4907-9535-0ce4ef46383c" volumeName="kubernetes.io/empty-dir/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-textfile" seLinuxMountContext="" Oct 31 17:27:58.338082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338035 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a5dd3d-3ace-4a01-b358-d418d1fa81aa" volumeName="kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 31 17:27:58.338082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338047 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="511ee639-1612-4e36-812e-70f7865291c1" volumeName="kubernetes.io/empty-dir/511ee639-1612-4e36-812e-70f7865291c1-tmp" seLinuxMountContext="" Oct 31 17:27:58.338082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338064 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64a53072-d4ff-45d1-ab01-249f63874e30" volumeName="kubernetes.io/configmap/64a53072-d4ff-45d1-ab01-249f63874e30-multus-daemon-config" seLinuxMountContext="" Oct 31 17:27:58.338082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338079 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-db" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338092 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="016aec04-e1a2-4dd4-8a59-9cfc0418da6f" volumeName="kubernetes.io/projected/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-kube-api-access-b9kkw" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338106 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="016aec04-e1a2-4dd4-8a59-9cfc0418da6f" volumeName="kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338118 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-kube-api-access-q2mw5" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338143 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b4b2af-0bd0-4907-9535-0ce4ef46383c" volumeName="kubernetes.io/configmap/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338157 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-config-out" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338170 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="511ee639-1612-4e36-812e-70f7865291c1" volumeName="kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338182 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="795fdffb-4631-44b0-a0fa-91b74548f1e9" volumeName="kubernetes.io/configmap/795fdffb-4631-44b0-a0fa-91b74548f1e9-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338205 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="795fdffb-4631-44b0-a0fa-91b74548f1e9" volumeName="kubernetes.io/projected/795fdffb-4631-44b0-a0fa-91b74548f1e9-kube-api-access-w9tw2" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338217 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9206f5fc-3e4d-4574-8c55-aaca1a33e3d4" volumeName="kubernetes.io/projected/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-api-access-jx2wp" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338230 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b4b2af-0bd0-4907-9535-0ce4ef46383c" volumeName="kubernetes.io/secret/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-tls" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338243 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="511ee639-1612-4e36-812e-70f7865291c1" volumeName="kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338271 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a5dd3d-3ace-4a01-b358-d418d1fa81aa" volumeName="kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-secret-metrics-server-tls" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338300 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64a53072-d4ff-45d1-ab01-249f63874e30" volumeName="kubernetes.io/configmap/64a53072-d4ff-45d1-ab01-249f63874e30-cni-binary-copy" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338313 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e895668-db69-4458-9e3d-fcd82e31084b" volumeName="kubernetes.io/secret/6e895668-db69-4458-9e3d-fcd82e31084b-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338326 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806f6a75-275f-48d2-ae95-03e1a5e46515" volumeName="kubernetes.io/projected/806f6a75-275f-48d2-ae95-03e1a5e46515-kube-api-access-wwxk9" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338338 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338352 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-kube-api-access-zkkrp" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338366 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9206f5fc-3e4d-4574-8c55-aaca1a33e3d4" volumeName="kubernetes.io/empty-dir/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-volume-directive-shadow" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338378 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd29fa4f-f33c-4c02-988c-afcfb47b6ca0" volumeName="kubernetes.io/projected/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-kube-api-access-x7s8n" seLinuxMountContext="" Oct 31 17:27:58.338393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338393 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="016aec04-e1a2-4dd4-8a59-9cfc0418da6f" volumeName="kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-tls" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338406 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a5dd3d-3ace-4a01-b358-d418d1fa81aa" volumeName="kubernetes.io/configmap/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338419 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76940cd0-ef63-4e10-965e-e871ab476281" volumeName="kubernetes.io/configmap/76940cd0-ef63-4e10-965e-e871ab476281-konnectivity-ca" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338443 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338456 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d30a63cf-3116-466f-8e4d-c77f6eaacb37" volumeName="kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338468 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338481 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2269cc9c-7732-4144-bc2e-8dafebdd1957" volumeName="kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-federate-client-tls" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338493 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="511ee639-1612-4e36-812e-70f7865291c1" volumeName="kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338507 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fa107ff-c9d8-41a8-9baa-190bd8714606" volumeName="kubernetes.io/projected/6fa107ff-c9d8-41a8-9baa-190bd8714606-kube-api-access-rdphq" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338520 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76940cd0-ef63-4e10-965e-e871ab476281" volumeName="kubernetes.io/secret/76940cd0-ef63-4e10-965e-e871ab476281-agent-certs" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338532 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a3fc242-f912-4cdd-8f92-dcd8319ba40a" volumeName="kubernetes.io/secret/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338544 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9206f5fc-3e4d-4574-8c55-aaca1a33e3d4" volumeName="kubernetes.io/secret/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338556 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338568 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-cluster-tls-config" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338582 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e895668-db69-4458-9e3d-fcd82e31084b" volumeName="kubernetes.io/secret/6e895668-db69-4458-9e3d-fcd82e31084b-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338594 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fa107ff-c9d8-41a8-9baa-190bd8714606" volumeName="kubernetes.io/secret/6fa107ff-c9d8-41a8-9baa-190bd8714606-cert" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338605 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338618 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a5dd3d-3ace-4a01-b358-d418d1fa81aa" volumeName="kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-client-ca-bundle" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338631 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2269cc9c-7732-4144-bc2e-8dafebdd1957" volumeName="kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338644 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338655 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="795fdffb-4631-44b0-a0fa-91b74548f1e9" volumeName="kubernetes.io/secret/795fdffb-4631-44b0-a0fa-91b74548f1e9-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338676 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806f6a75-275f-48d2-ae95-03e1a5e46515" volumeName="kubernetes.io/configmap/806f6a75-275f-48d2-ae95-03e1a5e46515-serviceca" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338686 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9206f5fc-3e4d-4574-8c55-aaca1a33e3d4" volumeName="kubernetes.io/secret/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338698 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd29fa4f-f33c-4c02-988c-afcfb47b6ca0" volumeName="kubernetes.io/secret/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-metrics-tls" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338710 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="016aec04-e1a2-4dd4-8a59-9cfc0418da6f" volumeName="kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338721 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-metrics-client-ca" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338735 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2269cc9c-7732-4144-bc2e-8dafebdd1957" volumeName="kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338747 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44b0b68e-a3e0-4bb7-b987-92b7dacd0612" volumeName="kubernetes.io/empty-dir/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-tmp" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338759 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a5dd3d-3ace-4a01-b358-d418d1fa81aa" volumeName="kubernetes.io/configmap/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-metrics-server-audit-profiles" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338778 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="795fdffb-4631-44b0-a0fa-91b74548f1e9" volumeName="kubernetes.io/empty-dir/795fdffb-4631-44b0-a0fa-91b74548f1e9-data-volume" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338793 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-main-db" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338804 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="159b2703-3b66-4ddc-8762-b4546b942282" volumeName="kubernetes.io/projected/159b2703-3b66-4ddc-8762-b4546b942282-kube-api-access-c8d9h" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338816 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-tls-assets" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338828 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-config-volume" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338841 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="511ee639-1612-4e36-812e-70f7865291c1" volumeName="kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338853 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b4b2af-0bd0-4907-9535-0ce4ef46383c" volumeName="kubernetes.io/projected/93b4b2af-0bd0-4907-9535-0ce4ef46383c-kube-api-access-hh6wh" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338883 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="016aec04-e1a2-4dd4-8a59-9cfc0418da6f" volumeName="kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338896 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a5dd3d-3ace-4a01-b358-d418d1fa81aa" volumeName="kubernetes.io/empty-dir/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-audit-log" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338907 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338921 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd29fa4f-f33c-4c02-988c-afcfb47b6ca0" volumeName="kubernetes.io/empty-dir/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-tmp-dir" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338933 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-web-config" seLinuxMountContext="" Oct 31 17:27:58.338934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338945 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2269cc9c-7732-4144-bc2e-8dafebdd1957" volumeName="kubernetes.io/projected/2269cc9c-7732-4144-bc2e-8dafebdd1957-kube-api-access-vtqcd" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338958 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b4b2af-0bd0-4907-9535-0ce4ef46383c" volumeName="kubernetes.io/secret/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338969 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-grpc-tls" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338982 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a5dd3d-3ace-4a01-b358-d418d1fa81aa" volumeName="kubernetes.io/projected/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-kube-api-access-9plqg" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.338995 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2269cc9c-7732-4144-bc2e-8dafebdd1957" volumeName="kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339008 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2269cc9c-7732-4144-bc2e-8dafebdd1957" volumeName="kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-telemeter-client-tls" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339020 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1" volumeName="kubernetes.io/projected/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-kube-api-access-nrmhn" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339031 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-metrics-client-certs" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339043 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2269cc9c-7732-4144-bc2e-8dafebdd1957" volumeName="kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-metrics-client-ca" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339056 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="016409a4-3533-46e9-970d-4154787d4924" volumeName="kubernetes.io/projected/016409a4-3533-46e9-970d-4154787d4924-kube-api-access-bldpx" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339069 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="016aec04-e1a2-4dd4-8a59-9cfc0418da6f" volumeName="kubernetes.io/configmap/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-metrics-client-ca" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339081 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-config-out" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339093 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a3fc242-f912-4cdd-8f92-dcd8319ba40a" volumeName="kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovnkube-config" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339105 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1" volumeName="kubernetes.io/configmap/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-iptables-alerter-script" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339117 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44b0b68e-a3e0-4bb7-b987-92b7dacd0612" volumeName="kubernetes.io/projected/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-kube-api-access-blr7j" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339128 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a3fc242-f912-4cdd-8f92-dcd8319ba40a" volumeName="kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovnkube-script-lib" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339140 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339152 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339163 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339175 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" volumeName="kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339186 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="016aec04-e1a2-4dd4-8a59-9cfc0418da6f" volumeName="kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339199 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="176a27de-94c3-482d-9bcd-e335ba12400a" volumeName="kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339212 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64a53072-d4ff-45d1-ab01-249f63874e30" volumeName="kubernetes.io/projected/64a53072-d4ff-45d1-ab01-249f63874e30-kube-api-access-txcnm" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339224 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a3fc242-f912-4cdd-8f92-dcd8319ba40a" volumeName="kubernetes.io/projected/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-kube-api-access-qq4bl" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339236 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-metrics-client-ca" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339247 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339275 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339287 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d30a63cf-3116-466f-8e4d-c77f6eaacb37" volumeName="kubernetes.io/projected/d30a63cf-3116-466f-8e4d-c77f6eaacb37-kube-api-access-n4x98" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339300 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" volumeName="kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cni-binary-copy" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339311 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="016aec04-e1a2-4dd4-8a59-9cfc0418da6f" volumeName="kubernetes.io/empty-dir/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-tmp" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339323 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="511ee639-1612-4e36-812e-70f7865291c1" volumeName="kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339334 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="016aec04-e1a2-4dd4-8a59-9cfc0418da6f" volumeName="kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-grpc-tls" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339345 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9206f5fc-3e4d-4574-8c55-aaca1a33e3d4" volumeName="kubernetes.io/configmap/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-metrics-client-ca" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339357 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339370 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2269cc9c-7732-4144-bc2e-8dafebdd1957" volumeName="kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-secret-telemeter-client" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339382 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd2a7b5c-fbe5-43d3-9121-403946f33aca" volumeName="kubernetes.io/secret/bd2a7b5c-fbe5-43d3-9121-403946f33aca-monitoring-plugin-cert" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339413 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b4b2af-0bd0-4907-9535-0ce4ef46383c" volumeName="kubernetes.io/configmap/93b4b2af-0bd0-4907-9535-0ce4ef46383c-metrics-client-ca" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339426 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-web-config" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339437 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bec5b1da-4675-4713-9e51-9e7be00b5ba0" volumeName="kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339447 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="159b2703-3b66-4ddc-8762-b4546b942282" volumeName="kubernetes.io/empty-dir/159b2703-3b66-4ddc-8762-b4546b942282-tmp-dir" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339461 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9206f5fc-3e4d-4574-8c55-aaca1a33e3d4" volumeName="kubernetes.io/configmap/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339473 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339485 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b556c44-090c-4e81-9d8d-2fac4026b0df" volumeName="kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-tls-assets" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339505 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd29fa4f-f33c-4c02-988c-afcfb47b6ca0" volumeName="kubernetes.io/configmap/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-config-volume" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339517 7618 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd2a7b5c-fbe5-43d3-9121-403946f33aca" volumeName="kubernetes.io/projected/bd2a7b5c-fbe5-43d3-9121-403946f33aca-kube-api-access-wmv4j" seLinuxMountContext="" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339529 7618 reconstruct.go:97] "Volume reconstruction finished" Oct 31 17:27:58.340324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.339537 7618 reconciler.go:29] "Reconciler: start to sync state" Oct 31 17:27:58.364226 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.364170 7618 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 17:27:58.365702 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.365675 7618 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 17:27:58.365702 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.365701 7618 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 17:27:58.365886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.365724 7618 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 17:27:58.365886 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:58.365768 7618 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 17:27:58.368102 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.368081 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:27:58.370365 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.370335 7618 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 17:27:58.370606 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.370581 7618 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 17:27:58.372240 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.372215 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:27:58.379386 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.379362 7618 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 17:27:58.389608 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.389578 7618 generic.go:356] "Generic (PLEG): container finished" podID="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" containerID="4b7ed9aceb2a4abb6ae92b1258af2be9354ce3315e431cb6ed7b33778b1aed35" exitCode=0 Oct 31 17:27:58.389608 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.389603 7618 generic.go:356] "Generic (PLEG): container finished" podID="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" containerID="d9365445890285d27c95b42868231e28b3d95ba734d7437b21dfd911e4fefb5f" exitCode=0 Oct 31 17:27:58.389608 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.389610 7618 generic.go:356] "Generic (PLEG): container finished" podID="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" containerID="96e8d1eefc7a13464ca873d4dc51d06d5304f325380796ac6c4a4d12991cbb8f" exitCode=0 Oct 31 17:27:58.389608 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.389615 7618 generic.go:356] "Generic (PLEG): container finished" podID="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" containerID="10fe61c8214d628e537283995b86a2e0685ff547968a13761602a99d96ac5b62" exitCode=0 Oct 31 17:27:58.389895 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.389621 7618 generic.go:356] "Generic (PLEG): container finished" podID="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" containerID="2a8a0e6f3b3a7cc7c8ba024d862a25e7707045d45132d3afcbe49b89a1aa180b" exitCode=0 Oct 31 17:27:58.389895 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.389626 7618 generic.go:356] "Generic (PLEG): container finished" podID="4b8b80aa-d5dc-41cd-8b89-5a45afc4064f" containerID="13b0da64d7d862384fad9d3d165ed348d769dcdf6f37794e66270033ee8e51e0" exitCode=0 Oct 31 17:27:58.390094 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.390077 7618 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 17:27:58.390884 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.390863 7618 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 17:27:58.398221 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.398192 7618 generic.go:356] "Generic (PLEG): container finished" podID="93b4b2af-0bd0-4907-9535-0ce4ef46383c" containerID="dcc90125ec09232730524eef742eb9daefc29ce81987c28639a227aaf7c5c462" exitCode=0 Oct 31 17:27:58.407118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.407090 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:27:58.407519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.407487 7618 generic.go:356] "Generic (PLEG): container finished" podID="7a3fc242-f912-4cdd-8f92-dcd8319ba40a" containerID="89272db701614577b68dff17143a65604ce33b86a053ab6084beee0095c46dab" exitCode=1 Oct 31 17:27:58.412726 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.412690 7618 generic.go:356] "Generic (PLEG): container finished" podID="8765d5fdccb7ed0c5cd03e206e2ad979" containerID="e8f1375616f561f3e093208f842939d4847fb5e18971a6d69a89bc3b3d753fe1" exitCode=0 Oct 31 17:27:58.417857 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.417836 7618 generic.go:356] "Generic (PLEG): container finished" podID="176a27de-94c3-482d-9bcd-e335ba12400a" containerID="05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b" exitCode=0 Oct 31 17:27:58.427362 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.427329 7618 generic.go:356] "Generic (PLEG): container finished" podID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerID="c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2" exitCode=0 Oct 31 17:27:58.465893 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:58.465819 7618 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 17:27:58.466784 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.466756 7618 manager.go:324] Recovery completed Oct 31 17:27:58.493659 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.493628 7618 cpu_manager.go:226] "Starting" policy="none" Oct 31 17:27:58.493659 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.493650 7618 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 17:27:58.493908 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.493679 7618 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 17:27:58.493908 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.493846 7618 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 17:27:58.493908 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.493861 7618 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 17:27:58.493908 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.493886 7618 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 17:27:58.493908 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.493895 7618 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 17:27:58.493908 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.493902 7618 policy_none.go:50] "Start" Oct 31 17:27:58.494181 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.493916 7618 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 17:27:58.494181 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.493929 7618 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 17:27:58.494181 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.494110 7618 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 17:27:58.494181 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.494121 7618 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 17:27:58.494181 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.494129 7618 policy_none.go:44] "Start" Oct 31 17:27:58.499224 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.499189 7618 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 17:27:58.501851 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.501833 7618 manager.go:341] "Starting Device Plugin manager" Oct 31 17:27:58.502340 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:58.501861 7618 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 17:27:58.502403 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.502342 7618 server.go:87] "Starting device plugin registration server" Oct 31 17:27:58.502620 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.502609 7618 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 17:27:58.502652 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.502620 7618 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 17:27:58.502745 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.502724 7618 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 17:27:58.502871 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.502837 7618 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 17:27:58.502911 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.502883 7618 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 17:27:58.502911 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.502890 7618 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 17:27:58.503000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.502931 7618 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T17:27:58.502869572Z","UUID":"f1118d18-02ae-4a9e-826b-8054645a9870","Handler":null,"Name":"","Endpoint":""} Oct 31 17:27:58.505610 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:58.505576 7618 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 17:27:58.505752 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.505598 7618 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 17:27:58.505752 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.505737 7618 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 17:27:58.603183 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.603142 7618 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 17:27:58.606745 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.606721 7618 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientMemory" Oct 31 17:27:58.606875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.606760 7618 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 17:27:58.606875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.606782 7618 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientPID" Oct 31 17:27:58.606875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.606850 7618 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.635507 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.635475 7618 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.635640 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.635557 7618 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.666469 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.666398 7618 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal"] Oct 31 17:27:58.667110 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.667076 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" event={"ID":"8389bef8c4009e269aafe36de827cd22","Type":"ContainerStarted","Data":"a7a445a65e3e2e610c575addbd63e9910a314bd935403af563a36d8982bb4947"} Oct 31 17:27:58.667207 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.667115 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" event={"ID":"8389bef8c4009e269aafe36de827cd22","Type":"ContainerStarted","Data":"d5989dbdb2d1c5d28a2b391e91e571a62d76150c1afe36bf4435b89aedd32b6c"} Oct 31 17:27:58.667207 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.667165 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" event={"ID":"8765d5fdccb7ed0c5cd03e206e2ad979","Type":"ContainerStarted","Data":"d8d3cea8073073b0b0e25daa97b883b3f1c563309bff1ea1c8101e786556e371"} Oct 31 17:27:58.667207 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.667177 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" event={"ID":"8765d5fdccb7ed0c5cd03e206e2ad979","Type":"ContainerDied","Data":"e8f1375616f561f3e093208f842939d4847fb5e18971a6d69a89bc3b3d753fe1"} Oct 31 17:27:58.667207 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.667191 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" event={"ID":"8765d5fdccb7ed0c5cd03e206e2ad979","Type":"ContainerStarted","Data":"49daaf0fda866993155a9e9a1e9d051c758c64ba9cde917ea68014e7b17ca97a"} Oct 31 17:27:58.667455 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.667246 7618 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.667455 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.667331 7618 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.689406 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.689372 7618 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 17:27:58.689542 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:58.689430 7618 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-107.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.690026 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.690007 7618 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 17:27:58.690085 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:58.690044 7618 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.772460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.772381 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8765d5fdccb7ed0c5cd03e206e2ad979-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"8765d5fdccb7ed0c5cd03e206e2ad979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.772460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.772415 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8765d5fdccb7ed0c5cd03e206e2ad979-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"8765d5fdccb7ed0c5cd03e206e2ad979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.772460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.772441 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8389bef8c4009e269aafe36de827cd22-config\") pod \"kube-apiserver-proxy-ip-10-0-139-107.ec2.internal\" (UID: \"8389bef8c4009e269aafe36de827cd22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.872927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.872893 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8765d5fdccb7ed0c5cd03e206e2ad979-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"8765d5fdccb7ed0c5cd03e206e2ad979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.872927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.872927 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8765d5fdccb7ed0c5cd03e206e2ad979-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"8765d5fdccb7ed0c5cd03e206e2ad979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.873104 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.872954 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8389bef8c4009e269aafe36de827cd22-config\") pod \"kube-apiserver-proxy-ip-10-0-139-107.ec2.internal\" (UID: \"8389bef8c4009e269aafe36de827cd22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.873104 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.872998 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8765d5fdccb7ed0c5cd03e206e2ad979-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"8765d5fdccb7ed0c5cd03e206e2ad979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.873104 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.873016 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8765d5fdccb7ed0c5cd03e206e2ad979-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"8765d5fdccb7ed0c5cd03e206e2ad979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:27:58.873104 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:58.873075 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8389bef8c4009e269aafe36de827cd22-config\") pod \"kube-apiserver-proxy-ip-10-0-139-107.ec2.internal\" (UID: \"8389bef8c4009e269aafe36de827cd22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:27:59.319114 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.319072 7618 apiserver.go:52] "Watching apiserver" Oct 31 17:27:59.363827 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.363797 7618 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 17:27:59.364529 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.364501 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0","openshift-network-operator/iptables-alerter-cssnv","openshift-dns/node-resolver-58mtk","openshift-monitoring/node-exporter-66bdk","openshift-insights/insights-runtime-extractor-knkx9","openshift-image-registry/node-ca-9789n","openshift-multus/multus-additional-cni-plugins-2klfh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal","openshift-cluster-node-tuning-operator/tuned-cpz9p","openshift-ingress-canary/ingress-canary-x8mnz","openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw","openshift-monitoring/alertmanager-main-0","openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf","kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal","openshift-console/console-6f77577466-bjlt5","openshift-dns/dns-default-4jjks","openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv","kube-system/global-pull-secret-syncer-swbqv","openshift-multus/multus-862p4","openshift-multus/network-metrics-daemon-9zfqw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4","openshift-ovn-kubernetes/ovnkube-node-tdrsm","openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk","openshift-network-diagnostics/network-check-target-cd9s8","kube-system/konnectivity-agent-v9fs8","openshift-monitoring/metrics-server-5db8c4796b-lwk5p","openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v"] Oct 31 17:27:59.378018 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.377988 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 17:27:59.378170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378025 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 17:27:59.378170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378071 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 17:27:59.378170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378114 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 17:27:59.378318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378172 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 17:27:59.378416 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378396 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 17:27:59.378476 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378444 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vvnlg\"" Oct 31 17:27:59.378476 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378455 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k6pvq\"" Oct 31 17:27:59.378538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378499 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 17:27:59.378538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378529 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x4c5q\"" Oct 31 17:27:59.378597 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378546 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nk5tf\"" Oct 31 17:27:59.378627 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378600 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 17:27:59.378709 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378693 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 17:27:59.378871 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378854 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 17:27:59.378933 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378881 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zv4z8\"" Oct 31 17:27:59.378933 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378898 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 17:27:59.379000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378979 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 17:27:59.379032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.378996 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 17:27:59.436073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.436043 7618 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 17:27:59.443189 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.443160 7618 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:27:59.443340 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.443317 7618 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:27:59.449754 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.449730 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 17:27:59.449867 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.449778 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 17:27:59.449867 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.449834 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 17:27:59.449867 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.449838 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4g54t\"" Oct 31 17:27:59.449867 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.449864 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 17:27:59.450073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.449888 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 17:27:59.450073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.449918 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 17:27:59.450073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.449948 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 17:27:59.450073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.449888 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tj66q\"" Oct 31 17:27:59.450073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.449990 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dm9tc\"" Oct 31 17:27:59.450073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450000 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 17:27:59.450073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450022 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 17:27:59.450073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450046 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 17:27:59.450073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450053 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5rzbg\"" Oct 31 17:27:59.450486 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450108 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 17:27:59.450533 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450507 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 17:27:59.450533 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450514 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 17:27:59.450533 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450517 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 17:27:59.450692 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450564 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 17:27:59.450692 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450566 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mcfpr\"" Oct 31 17:27:59.450692 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450608 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 17:27:59.450692 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450615 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 17:27:59.450692 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450662 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 17:27:59.450692 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450664 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5bhk8\"" Oct 31 17:27:59.450692 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450678 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dre6cbb61jri1\"" Oct 31 17:27:59.451013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450683 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 17:27:59.451013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450684 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 17:27:59.451013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450753 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 17:27:59.451013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450763 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8d7mp\"" Oct 31 17:27:59.451013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450772 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g2t4b\"" Oct 31 17:27:59.451013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450817 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9iap3qsv6ukg3\"" Oct 31 17:27:59.451013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450849 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 17:27:59.451013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450916 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 17:27:59.451345 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.450918 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 17:27:59.451389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.451371 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4pmd9\"" Oct 31 17:27:59.451436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.451392 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rr8f2\"" Oct 31 17:27:59.451588 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.451572 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 17:27:59.451717 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.451700 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 17:27:59.451813 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.451798 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 17:27:59.451860 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.451819 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qxdq5\"" Oct 31 17:27:59.451940 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.451926 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z2hdt\"" Oct 31 17:27:59.451984 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.451938 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 17:27:59.452110 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.451804 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 17:27:59.452162 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.452148 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6jpgp\"" Oct 31 17:27:59.452380 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.452333 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2xgtr\"" Oct 31 17:27:59.452432 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.452417 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 17:27:59.452478 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.452466 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8gjvp\"" Oct 31 17:27:59.452620 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.452604 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9gvm6\"" Oct 31 17:27:59.452679 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.452626 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 17:27:59.452679 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.452661 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 17:27:59.452762 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.452709 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 17:27:59.452762 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.452726 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 17:27:59.452951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.452937 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hzhgj\"" Oct 31 17:27:59.453051 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.453035 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xbzzg\"" Oct 31 17:27:59.453103 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.453056 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 17:27:59.453103 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.453067 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ccmlejc9fapc\"" Oct 31 17:27:59.453187 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.453059 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 17:27:59.453187 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.453169 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 17:27:59.453443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.453427 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 17:27:59.453538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.453520 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 17:27:59.453652 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.453635 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 17:27:59.453777 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.453759 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5hshp\"" Oct 31 17:27:59.453826 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.453649 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 17:27:59.453941 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.453920 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 17:27:59.463487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.463467 7618 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 17:27:59.463619 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.463514 7618 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-107.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Oct 31 17:27:59.464127 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.464110 7618 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 17:27:59.464175 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.464148 7618 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Oct 31 17:27:59.476562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.476532 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-cni-netd\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.476700 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.476566 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fa107ff-c9d8-41a8-9baa-190bd8714606-cert\") pod \"ingress-canary-x8mnz\" (UID: \"6fa107ff-c9d8-41a8-9baa-190bd8714606\") " pod="openshift-ingress-canary/ingress-canary-x8mnz" Oct 31 17:27:59.476700 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.476592 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64a53072-d4ff-45d1-ab01-249f63874e30-multus-daemon-config\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.476700 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.476611 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.476700 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.476673 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-systemd\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.476856 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.476714 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-secret-telemeter-client\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.476856 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.476741 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.476922 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.476865 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.476985 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.476963 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-cnibin\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.476985 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.476977 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6fa107ff-c9d8-41a8-9baa-190bd8714606-cert\") pod \"ingress-canary-x8mnz\" (UID: \"6fa107ff-c9d8-41a8-9baa-190bd8714606\") " pod="openshift-ingress-canary/ingress-canary-x8mnz" Oct 31 17:27:59.477062 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.476986 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-secret-telemeter-client\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.477062 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477016 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-hostroot\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.477062 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477052 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.477203 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477071 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64a53072-d4ff-45d1-ab01-249f63874e30-multus-daemon-config\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.477203 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477077 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.477203 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.477128 7618 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 17:27:59.477203 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477127 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-tls\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.477203 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477179 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-cni-bin\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.477203 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477199 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-tuned\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.477549 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.477299 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:59.977244977 +0000 UTC m=+1.725219448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 17:27:59.477549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477308 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-tuned\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.477549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477332 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-tls\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.477549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477330 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-cni-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.477549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477377 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-secret-metrics-server-tls\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.477549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477427 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.477549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477493 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysctl-conf\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.477549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477509 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cnibin\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.477549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477527 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx2wp\" (UniqueName: \"kubernetes.io/projected/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-api-access-jx2wp\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:59.477970 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477558 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.477970 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477606 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.477970 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477640 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-secret-metrics-server-tls\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.477970 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477666 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwxk9\" (UniqueName: \"kubernetes.io/projected/806f6a75-275f-48d2-ae95-03e1a5e46515-kube-api-access-wwxk9\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:27:59.477970 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.477727 7618 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 17:27:59.477970 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.477805 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:59.97778662 +0000 UTC m=+1.725761078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-oauth-config" not registered Oct 31 17:27:59.477970 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477891 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.478334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.477985 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-systemd-units\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.478334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478020 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.478334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478063 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bec5b1da-4675-4713-9e51-9e7be00b5ba0-dbus\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:59.478334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478110 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdphq\" (UniqueName: \"kubernetes.io/projected/6fa107ff-c9d8-41a8-9baa-190bd8714606-kube-api-access-rdphq\") pod \"ingress-canary-x8mnz\" (UID: \"6fa107ff-c9d8-41a8-9baa-190bd8714606\") " pod="openshift-ingress-canary/ingress-canary-x8mnz" Oct 31 17:27:59.478334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478113 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.478334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478236 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 17:27:59.478334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478281 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795fdffb-4631-44b0-a0fa-91b74548f1e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:59.478334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478318 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9kkw\" (UniqueName: \"kubernetes.io/projected/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-kube-api-access-b9kkw\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478344 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478351 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478356 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478373 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blr7j\" (UniqueName: \"kubernetes.io/projected/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-kube-api-access-blr7j\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478343 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478397 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478422 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s8n\" (UniqueName: \"kubernetes.io/projected/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-kube-api-access-x7s8n\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478376 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478445 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.478454 7618 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478469 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478498 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-lib-modules\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.478521 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:59.978506599 +0000 UTC m=+1.726481050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-serving-cert" not registered Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478531 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478558 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-ovn\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478634 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-log-socket\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478668 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-accelerators-collector-config\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478677 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 17:27:59.478714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478719 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-k8s-cni-cncf-io\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478753 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4x98\" (UniqueName: \"kubernetes.io/projected/d30a63cf-3116-466f-8e4d-c77f6eaacb37-kube-api-access-n4x98\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478798 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30a63cf-3116-466f-8e4d-c77f6eaacb37-metrics-certs\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478796 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478820 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478908 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478917 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478917 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478932 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478928 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.478981 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-multus-certs\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479008 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-var-lib-kubelet\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479058 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e895668-db69-4458-9e3d-fcd82e31084b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479091 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-system-cni-dir\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479121 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9tw2\" (UniqueName: \"kubernetes.io/projected/795fdffb-4631-44b0-a0fa-91b74548f1e9-kube-api-access-w9tw2\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479151 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479163 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479178 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479206 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-systemd\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479226 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479230 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-cni-bin\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479272 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-registration-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479300 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmv4j\" (UniqueName: \"kubernetes.io/projected/bd2a7b5c-fbe5-43d3-9121-403946f33aca-kube-api-access-wmv4j\") pod \"monitoring-plugin-77f4f76594-t2v2v\" (UID: \"bd2a7b5c-fbe5-43d3-9121-403946f33aca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479344 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-tmp-dir\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479373 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479413 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479467 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-tls\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479497 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-tmp-dir\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479547 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-audit-log\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479607 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-config\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.479615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479633 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-audit-log\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479667 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479681 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-tls\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479684 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/795fdffb-4631-44b0-a0fa-91b74548f1e9-data-volume\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.479719 7618 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479735 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/795fdffb-4631-44b0-a0fa-91b74548f1e9-data-volume\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479741 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/159b2703-3b66-4ddc-8762-b4546b942282-hosts-file\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.479759 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:59.97974982 +0000 UTC m=+1.727724267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"service-ca" not registered Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479849 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479878 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479894 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkrp\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-kube-api-access-zkkrp\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479940 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.479980 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480027 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480163 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480171 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480180 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480187 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-config\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480196 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480246 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480271 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480276 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480300 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480319 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480330 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-host\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480359 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovnkube-script-lib\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480385 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480400 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-serving-certs-ca-bundle\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480430 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480442 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480455 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-kubelet\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480478 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480482 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-web-config\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480499 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480587 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-serving-certs-ca-bundle\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480600 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-secret-metrics-server-client-certs\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480628 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76940cd0-ef63-4e10-965e-e871ab476281-agent-certs\") pod \"konnectivity-agent-v9fs8\" (UID: \"76940cd0-ef63-4e10-965e-e871ab476281\") " pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480674 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480676 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480679 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-web-config\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480698 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480718 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480728 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480747 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480832 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-modprobe-d\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480837 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-secret-metrics-server-client-certs\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480867 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovnkube-config\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480885 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76940cd0-ef63-4e10-965e-e871ab476281-agent-certs\") pod \"konnectivity-agent-v9fs8\" (UID: \"76940cd0-ef63-4e10-965e-e871ab476281\") " pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480887 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-sys-fs\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480949 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.480966 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 17:27:59.480997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481017 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-telemeter-client-tls\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481044 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrmhn\" (UniqueName: \"kubernetes.io/projected/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-kube-api-access-nrmhn\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481076 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481098 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-client-ca-bundle\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481098 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovnkube-config\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481118 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e895668-db69-4458-9e3d-fcd82e31084b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481161 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93b4b2af-0bd0-4907-9535-0ce4ef46383c-metrics-client-ca\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481188 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481217 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481242 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159b2703-3b66-4ddc-8762-b4546b942282-tmp-dir\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481285 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/806f6a75-275f-48d2-ae95-03e1a5e46515-serviceca\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481297 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-telemeter-client-tls\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481312 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481334 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481346 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9plqg\" (UniqueName: \"kubernetes.io/projected/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-kube-api-access-9plqg\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481374 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-run\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481399 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481430 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldpx\" (UniqueName: \"kubernetes.io/projected/016409a4-3533-46e9-970d-4154787d4924-kube-api-access-bldpx\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481428 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93b4b2af-0bd0-4907-9535-0ce4ef46383c-metrics-client-ca\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481429 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/159b2703-3b66-4ddc-8762-b4546b942282-tmp-dir\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481489 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481489 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481513 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-host-slash\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481544 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481546 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd2a7b5c-fbe5-43d3-9121-403946f33aca-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t2v2v\" (UID: \"bd2a7b5c-fbe5-43d3-9121-403946f33aca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481373 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-client-ca-bundle\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481603 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481618 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481660 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64a53072-d4ff-45d1-ab01-249f63874e30-cni-binary-copy\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481688 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-etc-kubernetes\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481666 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e895668-db69-4458-9e3d-fcd82e31084b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481727 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkrt\" (UniqueName: \"kubernetes.io/projected/6e895668-db69-4458-9e3d-fcd82e31084b-kube-api-access-8nkrt\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481769 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-slash\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481813 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64a53072-d4ff-45d1-ab01-249f63874e30-cni-binary-copy\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481847 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-node-log\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481883 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-device-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481888 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bd2a7b5c-fbe5-43d3-9121-403946f33aca-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t2v2v\" (UID: \"bd2a7b5c-fbe5-43d3-9121-403946f33aca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481912 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481960 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-metrics-client-ca\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481960 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/806f6a75-275f-48d2-ae95-03e1a5e46515-serviceca\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.481997 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.482010 7618 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482019 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-grpc-tls\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482040 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-netns\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482054 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-kubelet\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.482067 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:59.982052688 +0000 UTC m=+1.730027138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482098 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482131 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-conf-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482145 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-metrics-client-ca\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482177 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482215 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482226 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482276 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-kubernetes\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482294 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysctl-d\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482304 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482319 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-grpc-tls\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482313 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e895668-db69-4458-9e3d-fcd82e31084b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482353 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482439 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-run-netns\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482446 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482470 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-etc-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482510 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq4bl\" (UniqueName: \"kubernetes.io/projected/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-kube-api-access-qq4bl\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482520 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6e895668-db69-4458-9e3d-fcd82e31084b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482542 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-federate-client-tls\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482562 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46kf\" (UniqueName: \"kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf\") pod \"network-check-target-cd9s8\" (UID: \"bcd594b6-86d2-47ff-bc81-03bd655d3954\") " pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482577 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-tmp\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482593 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-config-out\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482609 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-var-lib-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482626 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482644 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482672 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-tmp\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482687 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2mw5\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-kube-api-access-q2mw5\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482716 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-config-out\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482775 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482817 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/511ee639-1612-4e36-812e-70f7865291c1-tmp\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482849 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqcd\" (UniqueName: \"kubernetes.io/projected/2269cc9c-7732-4144-bc2e-8dafebdd1957-kube-api-access-vtqcd\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482864 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482878 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/511ee639-1612-4e36-812e-70f7865291c1-tmp\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482886 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482818 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-federate-client-tls\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482879 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-sys\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482934 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/806f6a75-275f-48d2-ae95-03e1a5e46515-host\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482952 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482967 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.482986 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483010 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.483045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483046 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483126 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-metrics-tls\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483198 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-textfile\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483226 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483268 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-textfile\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483288 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483315 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483333 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483349 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-metrics-tls\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483351 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-wtmp\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483384 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-metrics-client-ca\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483544 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483567 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483602 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-iptables-alerter-script\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483612 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-metrics-client-ca\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483620 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bec5b1da-4675-4713-9e51-9e7be00b5ba0-original-pull-secret\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483633 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-system-cni-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483662 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdrvg\" (UniqueName: \"kubernetes.io/projected/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-kube-api-access-pdrvg\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483704 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483739 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/795fdffb-4631-44b0-a0fa-91b74548f1e9-crio-socket\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483770 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795fdffb-4631-44b0-a0fa-91b74548f1e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483786 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-iptables-alerter-script\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483799 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-root\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483826 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cni-binary-copy\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483878 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483913 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483920 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795fdffb-4631-44b0-a0fa-91b74548f1e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483922 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.483941 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8d9h\" (UniqueName: \"kubernetes.io/projected/159b2703-3b66-4ddc-8762-b4546b942282-kube-api-access-c8d9h\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484020 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-config-volume\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484032 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cni-binary-copy\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484054 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484071 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2269cc9c-7732-4144-bc2e-8dafebdd1957-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484082 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-tmp\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484090 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484110 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484137 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovn-node-metrics-cert\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484162 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-tmp\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484178 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484238 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-config-volume\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484277 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-metrics-server-audit-profiles\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484286 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484298 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484321 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484337 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484339 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bec5b1da-4675-4713-9e51-9e7be00b5ba0-kubelet-config\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.484290 7618 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484363 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-config-volume\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484383 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484394 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovn-node-metrics-cert\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.484399 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:59.984385084 +0000 UTC m=+1.732359547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-config" not registered Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484447 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-sys\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484467 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484509 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-metrics-server-audit-profiles\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484550 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-run-ovn-kubernetes\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484551 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484579 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-socket-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484587 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-config-volume\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484622 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-cni-multus\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484646 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txcnm\" (UniqueName: \"kubernetes.io/projected/64a53072-d4ff-45d1-ab01-249f63874e30-kube-api-access-txcnm\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484664 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484680 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484699 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484737 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-env-overrides\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484757 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6wh\" (UniqueName: \"kubernetes.io/projected/93b4b2af-0bd0-4907-9535-0ce4ef46383c-kube-api-access-hh6wh\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484764 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484777 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-socket-dir-parent\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484818 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484881 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76940cd0-ef63-4e10-965e-e871ab476281-konnectivity-ca\") pod \"konnectivity-agent-v9fs8\" (UID: \"76940cd0-ef63-4e10-965e-e871ab476281\") " pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484897 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484897 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-env-overrides\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484919 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysconfig\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484945 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.484949 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.485081 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76940cd0-ef63-4e10-965e-e871ab476281-konnectivity-ca\") pod \"konnectivity-agent-v9fs8\" (UID: \"76940cd0-ef63-4e10-965e-e871ab476281\") " pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:27:59.486081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.485174 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.489075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.489057 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.489198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.489175 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795fdffb-4631-44b0-a0fa-91b74548f1e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:59.489336 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.489320 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e895668-db69-4458-9e3d-fcd82e31084b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:59.489336 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.489326 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-accelerators-collector-config\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.491467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.490486 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 17:27:59.491467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.490556 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 17:27:59.491467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.490926 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 17:27:59.491467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.491164 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 17:27:59.491467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.491218 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-ovnkube-script-lib\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.492910 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.492888 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2269cc9c-7732-4144-bc2e-8dafebdd1957-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.493386 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.493371 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwxk9\" (UniqueName: \"kubernetes.io/projected/806f6a75-275f-48d2-ae95-03e1a5e46515-kube-api-access-wwxk9\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:27:59.493602 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.493588 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.493644 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.493619 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx2wp\" (UniqueName: \"kubernetes.io/projected/9206f5fc-3e4d-4574-8c55-aaca1a33e3d4-kube-api-access-jx2wp\") pod \"kube-state-metrics-5dc8d7d89b-spjzf\" (UID: \"9206f5fc-3e4d-4574-8c55-aaca1a33e3d4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-spjzf" Oct 31 17:27:59.494430 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.494413 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdphq\" (UniqueName: \"kubernetes.io/projected/6fa107ff-c9d8-41a8-9baa-190bd8714606-kube-api-access-rdphq\") pod \"ingress-canary-x8mnz\" (UID: \"6fa107ff-c9d8-41a8-9baa-190bd8714606\") " pod="openshift-ingress-canary/ingress-canary-x8mnz" Oct 31 17:27:59.494776 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.494754 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4x98\" (UniqueName: \"kubernetes.io/projected/d30a63cf-3116-466f-8e4d-c77f6eaacb37-kube-api-access-n4x98\") pod \"network-metrics-daemon-9zfqw\" (UID: \"d30a63cf-3116-466f-8e4d-c77f6eaacb37\") " pod="openshift-multus/network-metrics-daemon-9zfqw" Oct 31 17:27:59.494893 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.494879 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blr7j\" (UniqueName: \"kubernetes.io/projected/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-kube-api-access-blr7j\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.495663 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.495642 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7s8n\" (UniqueName: \"kubernetes.io/projected/bd29fa4f-f33c-4c02-988c-afcfb47b6ca0-kube-api-access-x7s8n\") pod \"dns-default-4jjks\" (UID: \"bd29fa4f-f33c-4c02-988c-afcfb47b6ca0\") " pod="openshift-dns/dns-default-4jjks" Oct 31 17:27:59.495887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.495870 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkrp\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-kube-api-access-zkkrp\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.495958 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.495907 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmv4j\" (UniqueName: \"kubernetes.io/projected/bd2a7b5c-fbe5-43d3-9121-403946f33aca-kube-api-access-wmv4j\") pod \"monitoring-plugin-77f4f76594-t2v2v\" (UID: \"bd2a7b5c-fbe5-43d3-9121-403946f33aca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:27:59.496162 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.496146 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9kkw\" (UniqueName: \"kubernetes.io/projected/016aec04-e1a2-4dd4-8a59-9cfc0418da6f-kube-api-access-b9kkw\") pod \"thanos-querier-6bf7478d4d-qqjlw\" (UID: \"016aec04-e1a2-4dd4-8a59-9cfc0418da6f\") " pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:27:59.496204 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.496147 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9tw2\" (UniqueName: \"kubernetes.io/projected/795fdffb-4631-44b0-a0fa-91b74548f1e9-kube-api-access-w9tw2\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:59.496399 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.496385 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrmhn\" (UniqueName: \"kubernetes.io/projected/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-kube-api-access-nrmhn\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:27:59.498163 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.498147 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldpx\" (UniqueName: \"kubernetes.io/projected/016409a4-3533-46e9-970d-4154787d4924-kube-api-access-bldpx\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.498671 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.498654 7618 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 17:27:59.498745 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.498677 7618 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 17:27:59.498745 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.498690 7618 projected.go:196] Error preparing data for projected volume kube-api-access-crxzn for pod openshift-console/console-6f77577466-bjlt5: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 17:27:59.498839 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.498775 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:27:59.998752771 +0000 UTC m=+1.746727219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-crxzn" (UniqueName: "kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 17:27:59.498877 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.498846 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq4bl\" (UniqueName: \"kubernetes.io/projected/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-kube-api-access-qq4bl\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.499168 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.499146 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9plqg\" (UniqueName: \"kubernetes.io/projected/58a5dd3d-3ace-4a01-b358-d418d1fa81aa-kube-api-access-9plqg\") pod \"metrics-server-5db8c4796b-lwk5p\" (UID: \"58a5dd3d-3ace-4a01-b358-d418d1fa81aa\") " pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:27:59.501050 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.499610 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkrt\" (UniqueName: \"kubernetes.io/projected/6e895668-db69-4458-9e3d-fcd82e31084b-kube-api-access-8nkrt\") pod \"openshift-state-metrics-7bd7489957-drfrv\" (UID: \"6e895668-db69-4458-9e3d-fcd82e31084b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-drfrv" Oct 31 17:27:59.501050 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.500724 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.505602 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.505171 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8d9h\" (UniqueName: \"kubernetes.io/projected/159b2703-3b66-4ddc-8762-b4546b942282-kube-api-access-c8d9h\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:27:59.505602 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.505365 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txcnm\" (UniqueName: \"kubernetes.io/projected/64a53072-d4ff-45d1-ab01-249f63874e30-kube-api-access-txcnm\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.505602 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.505450 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdrvg\" (UniqueName: \"kubernetes.io/projected/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-kube-api-access-pdrvg\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.505602 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.505555 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqcd\" (UniqueName: \"kubernetes.io/projected/2269cc9c-7732-4144-bc2e-8dafebdd1957-kube-api-access-vtqcd\") pod \"telemeter-client-6f6f8d64d6-sc4qk\" (UID: \"2269cc9c-7732-4144-bc2e-8dafebdd1957\") " pod="openshift-monitoring/telemeter-client-6f6f8d64d6-sc4qk" Oct 31 17:27:59.505602 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.505593 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g46kf\" (UniqueName: \"kubernetes.io/projected/bcd594b6-86d2-47ff-bc81-03bd655d3954-kube-api-access-g46kf\") pod \"network-check-target-cd9s8\" (UID: \"bcd594b6-86d2-47ff-bc81-03bd655d3954\") " pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:27:59.506085 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.506064 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6wh\" (UniqueName: \"kubernetes.io/projected/93b4b2af-0bd0-4907-9535-0ce4ef46383c-kube-api-access-hh6wh\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.506125 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.506104 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2mw5\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-kube-api-access-q2mw5\") pod \"alertmanager-main-0\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:27:59.506751 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.506733 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:27:59.586162 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586076 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bec5b1da-4675-4713-9e51-9e7be00b5ba0-kubelet-config\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:59.586162 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586113 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-sys\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.586162 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586138 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-run-ovn-kubernetes\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586164 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-socket-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586207 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-cni-multus\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586212 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bec5b1da-4675-4713-9e51-9e7be00b5ba0-kubelet-config\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586216 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-run-ovn-kubernetes\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586216 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-sys\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586234 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586297 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586305 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-socket-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586308 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-cni-multus\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586326 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-socket-dir-parent\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586349 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-socket-dir-parent\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586370 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysconfig\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586400 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-cni-netd\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586426 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysconfig\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586427 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-systemd\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586450 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-cni-netd\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586456 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-cnibin\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586483 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-cnibin\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586485 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-hostroot\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586499 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-systemd\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.586487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586506 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-hostroot\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586522 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-cni-bin\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586548 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-cni-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586568 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-cni-bin\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586574 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysctl-conf\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586600 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cnibin\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586611 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-cni-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586649 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-systemd-units\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586655 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-cnibin\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586669 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586700 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysctl-conf\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586702 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-systemd-units\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586703 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-etc-selinux\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586719 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bec5b1da-4675-4713-9e51-9e7be00b5ba0-dbus\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586750 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-lib-modules\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586772 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-ovn\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586794 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-log-socket\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586805 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bec5b1da-4675-4713-9e51-9e7be00b5ba0-dbus\") pod \"global-pull-secret-syncer-swbqv\" (UID: \"bec5b1da-4675-4713-9e51-9e7be00b5ba0\") " pod="kube-system/global-pull-secret-syncer-swbqv" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586816 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-k8s-cni-cncf-io\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586839 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-ovn\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586844 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-log-socket\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586842 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-multus-certs\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586890 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-lib-modules\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586891 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-k8s-cni-cncf-io\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586892 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-var-lib-kubelet\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586938 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-var-lib-kubelet\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586942 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-system-cni-dir\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586937 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-multus-certs\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586968 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-systemd\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586970 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-system-cni-dir\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587007 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-systemd\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587011 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-cni-bin\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.586987 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-cni-bin\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587042 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-registration-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587069 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/159b2703-3b66-4ddc-8762-b4546b942282-hosts-file\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587117 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/159b2703-3b66-4ddc-8762-b4546b942282-hosts-file\") pod \"node-resolver-58mtk\" (UID: \"159b2703-3b66-4ddc-8762-b4546b942282\") " pod="openshift-dns/node-resolver-58mtk" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587116 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-registration-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587134 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-host\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587162 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-kubelet\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587166 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-host\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587190 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-modprobe-d\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587207 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-sys-fs\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587238 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-run\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587276 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-var-lib-kubelet\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587284 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-sys-fs\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587285 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-host-slash\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:27:59.587273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587314 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-etc-kubernetes\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587330 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-slash\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587332 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-run\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587345 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-node-log\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587356 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-etc-kubernetes\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587363 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-device-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587386 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-device-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587391 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-node-log\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587313 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1-host-slash\") pod \"iptables-alerter-cssnv\" (UID: \"44e3309b-8a0f-42e7-bf62-a0d3d5a5b1a1\") " pod="openshift-network-operator/iptables-alerter-cssnv" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587347 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-modprobe-d\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587375 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-slash\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587408 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-netns\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587434 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-kubelet\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587466 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-kubelet\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587467 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-host-run-netns\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587482 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-conf-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587506 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-kubernetes\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587527 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysctl-d\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587543 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-run-netns\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587542 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-multus-conf-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587566 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-etc-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587581 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-kubernetes\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587593 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-var-lib-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587607 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-etc-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587612 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-var-lib-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587628 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44b0b68e-a3e0-4bb7-b987-92b7dacd0612-etc-sysctl-d\") pod \"tuned-cpz9p\" (UID: \"44b0b68e-a3e0-4bb7-b987-92b7dacd0612\") " pod="openshift-cluster-node-tuning-operator/tuned-cpz9p" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587632 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-sys\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587652 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-sys\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587662 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/806f6a75-275f-48d2-ae95-03e1a5e46515-host\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587603 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-host-run-netns\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587682 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587697 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/806f6a75-275f-48d2-ae95-03e1a5e46515-host\") pod \"node-ca-9789n\" (UID: \"806f6a75-275f-48d2-ae95-03e1a5e46515\") " pod="openshift-image-registry/node-ca-9789n" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587700 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587725 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-wtmp\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587745 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-system-cni-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587747 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b8b80aa-d5dc-41cd-8b89-5a45afc4064f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2klfh\" (UID: \"4b8b80aa-d5dc-41cd-8b89-5a45afc4064f\") " pod="openshift-multus/multus-additional-cni-plugins-2klfh" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587762 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/795fdffb-4631-44b0-a0fa-91b74548f1e9-crio-socket\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587777 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-root\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587787 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/016409a4-3533-46e9-970d-4154787d4924-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vbfh4\" (UID: \"016409a4-3533-46e9-970d-4154787d4924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vbfh4" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587794 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587797 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a53072-d4ff-45d1-ab01-249f63874e30-system-cni-dir\") pod \"multus-862p4\" (UID: \"64a53072-d4ff-45d1-ab01-249f63874e30\") " pod="openshift-multus/multus-862p4" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587803 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-node-exporter-wtmp\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587813 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/795fdffb-4631-44b0-a0fa-91b74548f1e9-crio-socket\") pod \"insights-runtime-extractor-knkx9\" (UID: \"795fdffb-4631-44b0-a0fa-91b74548f1e9\") " pod="openshift-insights/insights-runtime-extractor-knkx9" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587824 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93b4b2af-0bd0-4907-9535-0ce4ef46383c-root\") pod \"node-exporter-66bdk\" (UID: \"93b4b2af-0bd0-4907-9535-0ce4ef46383c\") " pod="openshift-monitoring/node-exporter-66bdk" Oct 31 17:27:59.588630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.587852 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a3fc242-f912-4cdd-8f92-dcd8319ba40a-run-openvswitch\") pod \"ovnkube-node-tdrsm\" (UID: \"7a3fc242-f912-4cdd-8f92-dcd8319ba40a\") " pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:27:59.992107 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.992068 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.992318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.992144 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.992318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.992188 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.992318 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.992197 7618 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 17:27:59.992318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.992219 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.992318 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.992288 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:00.992244357 +0000 UTC m=+2.740218823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 17:27:59.992564 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.992316 7618 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 17:27:59.992564 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.992333 7618 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 17:27:59.992564 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.992316 7618 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 17:27:59.992564 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.992375 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:00.992359072 +0000 UTC m=+2.740333521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 17:27:59.992564 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.992391 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:00.992383839 +0000 UTC m=+2.740358287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-oauth-config" not registered Oct 31 17:27:59.992564 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.992407 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.992564 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.992422 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:00.992404206 +0000 UTC m=+2.740378666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-config" not registered Oct 31 17:27:59.992564 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.992463 7618 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 17:27:59.992564 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.992504 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:00.992488079 +0000 UTC m=+2.740462550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-serving-cert" not registered Oct 31 17:27:59.992564 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:27:59.992524 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:27:59.992843 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.992615 7618 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 17:27:59.992843 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:27:59.992654 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:00.992644407 +0000 UTC m=+2.740618866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"service-ca" not registered Oct 31 17:28:00.093602 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:00.093563 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:00.093751 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:00.093729 7618 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 17:28:00.093809 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:00.093750 7618 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 17:28:00.093809 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:00.093761 7618 projected.go:196] Error preparing data for projected volume kube-api-access-crxzn for pod openshift-console/console-6f77577466-bjlt5: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 17:28:00.093873 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:00.093824 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:01.093808887 +0000 UTC m=+2.841783347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-crxzn" (UniqueName: "kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 17:28:01.001664 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:01.001616 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:01.001678 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:01.001722 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:01.001764 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.001769 7618 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.001828 7618 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.001829 7618 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.001848 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:03.001833878 +0000 UTC m=+4.749808339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-serving-cert" not registered Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.001861 7618 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.001885 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:03.001870403 +0000 UTC m=+4.749844868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-config" not registered Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.001899 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:03.001893315 +0000 UTC m=+4.749867760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"service-ca" not registered Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:01.001910 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.001933 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:03.001915464 +0000 UTC m=+4.749889928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.001939 7618 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.001969 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:03.001960153 +0000 UTC m=+4.749934612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:01.001982 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.002061 7618 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 17:28:01.002124 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.002096 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:03.002084778 +0000 UTC m=+4.750059236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-oauth-config" not registered Oct 31 17:28:01.102903 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:01.102866 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:01.103078 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.103026 7618 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 17:28:01.103078 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.103047 7618 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 17:28:01.103078 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.103057 7618 projected.go:196] Error preparing data for projected volume kube-api-access-crxzn for pod openshift-console/console-6f77577466-bjlt5: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 17:28:01.103167 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:01.103117 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:03.103104734 +0000 UTC m=+4.851079190 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-crxzn" (UniqueName: "kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 17:28:01.715935 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:01.715893 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:28:02.177504 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:02.177469 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2v2v" Oct 31 17:28:02.515488 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:02.515404 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4jjks" Oct 31 17:28:03.023237 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:03.023193 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:03.023237 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:03.023243 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:03.023528 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:03.023298 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:03.023528 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:03.023334 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:03.023528 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.023352 7618 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 17:28:03.023528 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:03.023381 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:03.023528 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.023435 7618 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 17:28:03.023528 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.023447 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:07.023425445 +0000 UTC m=+8.771399905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 17:28:03.023528 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.023451 7618 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 17:28:03.023528 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.023460 7618 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 17:28:03.023528 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:03.023493 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:03.023528 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.023499 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:07.023486278 +0000 UTC m=+8.771460737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-oauth-config" not registered Oct 31 17:28:03.023528 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.023525 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:07.023514883 +0000 UTC m=+8.771489330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-serving-cert" not registered Oct 31 17:28:03.023528 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.023528 7618 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 17:28:03.023882 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.023543 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:07.023534192 +0000 UTC m=+8.771508637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"service-ca" not registered Oct 31 17:28:03.023882 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.023459 7618 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 17:28:03.023882 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.023560 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:07.023551889 +0000 UTC m=+8.771526337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-config" not registered Oct 31 17:28:03.023882 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.023583 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:07.023569848 +0000 UTC m=+8.771544319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 17:28:03.124762 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:03.124722 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:03.124893 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.124852 7618 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 17:28:03.124893 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.124866 7618 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 17:28:03.124893 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.124876 7618 projected.go:196] Error preparing data for projected volume kube-api-access-crxzn for pod openshift-console/console-6f77577466-bjlt5: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 17:28:03.125008 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:03.124940 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:07.124927459 +0000 UTC m=+8.872901904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-crxzn" (UniqueName: "kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 17:28:03.814458 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:03.814425 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:28:04.266148 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:04.266116 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:04.444045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:04.444019 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:28:04.466146 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:04.466120 7618 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 17:28:04.583378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:04.583301 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bf7478d4d-qqjlw" Oct 31 17:28:07.070352 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:07.070310 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:07.070381 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:07.070415 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:07.070436 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:07.070461 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.070435 7618 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:07.070490 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.070496 7618 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.070468 7618 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.070532 7618 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.070551 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:15.070525868 +0000 UTC m=+16.818500330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.070568 7618 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.070569 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:15.070561839 +0000 UTC m=+16.818536284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-config" not registered Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.070601 7618 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.070611 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:15.070599847 +0000 UTC m=+16.818574292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.070621 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:15.070615748 +0000 UTC m=+16.818590194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-oauth-config" not registered Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.070631 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:15.070626 +0000 UTC m=+16.818600444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"service-ca" not registered Oct 31 17:28:07.070781 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.070641 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:15.070636454 +0000 UTC m=+16.818610899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-serving-cert" not registered Oct 31 17:28:07.171765 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:07.171722 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:07.171891 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.171851 7618 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 17:28:07.171891 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.171865 7618 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 17:28:07.171891 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.171877 7618 projected.go:196] Error preparing data for projected volume kube-api-access-crxzn for pod openshift-console/console-6f77577466-bjlt5: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 17:28:07.172005 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:07.171929 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:15.171913802 +0000 UTC m=+16.919888267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-crxzn" (UniqueName: "kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 17:28:08.578175 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:08.578135 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:28:10.254747 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:10.254664 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v9fs8" Oct 31 17:28:15.154518 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:15.154480 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:15.154518 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:15.154533 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:15.154556 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:15.154588 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.154626 7618 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:15.154631 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.154667 7618 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.154627 7618 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.154700 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:31.15467987 +0000 UTC m=+32.902654334 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.154701 7618 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.154732 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:31.154720453 +0000 UTC m=+32.902694899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"service-ca" not registered Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.154756 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:31.154742829 +0000 UTC m=+32.902717279 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-oauth-config" not registered Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.154707 7618 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.154810 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:31.154790705 +0000 UTC m=+32.902765160 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-config" not registered Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.154836 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:31.154829586 +0000 UTC m=+32.902804031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"console-serving-cert" not registered Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:15.154880 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.154943 7618 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 17:28:15.155149 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.154981 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:31.154966922 +0000 UTC m=+32.902941417 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 17:28:15.256004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:15.255963 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn\") pod \"console-6f77577466-bjlt5\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:15.256193 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.256148 7618 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 17:28:15.256193 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.256178 7618 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 17:28:15.256193 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.256190 7618 projected.go:196] Error preparing data for projected volume kube-api-access-crxzn for pod openshift-console/console-6f77577466-bjlt5: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 17:28:15.256594 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:15.256572 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn podName:511ee639-1612-4e36-812e-70f7865291c1 nodeName:}" failed. No retries permitted until 2026-10-31 17:28:31.256543676 +0000 UTC m=+33.004518138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-crxzn" (UniqueName: "kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn") pod "console-6f77577466-bjlt5" (UID: "511ee639-1612-4e36-812e-70f7865291c1") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 17:28:19.275539 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:19.275509 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5db8c4796b-lwk5p" Oct 31 17:28:24.395716 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.395672 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6f77577466-bjlt5" podUID="511ee639-1612-4e36-812e-70f7865291c1" containerName="console" containerID="cri-o://fbc6b961e24873f71116d543cefb5b67ceede670f46ec651fa891caa3a7065fe" gracePeriod=15 Oct 31 17:28:24.544026 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.543993 7618 generic.go:356] "Generic (PLEG): container finished" podID="511ee639-1612-4e36-812e-70f7865291c1" containerID="fbc6b961e24873f71116d543cefb5b67ceede670f46ec651fa891caa3a7065fe" exitCode=0 Oct 31 17:28:24.544179 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.544064 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f77577466-bjlt5" event={"ID":"511ee639-1612-4e36-812e-70f7865291c1","Type":"ContainerDied","Data":"fbc6b961e24873f71116d543cefb5b67ceede670f46ec651fa891caa3a7065fe"} Oct 31 17:28:24.661384 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.661305 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:24.768717 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.768681 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle\") pod \"511ee639-1612-4e36-812e-70f7865291c1\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " Oct 31 17:28:24.768902 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.768742 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511ee639-1612-4e36-812e-70f7865291c1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/511ee639-1612-4e36-812e-70f7865291c1-tmp\") pod \"511ee639-1612-4e36-812e-70f7865291c1\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " Oct 31 17:28:24.768902 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.768766 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca\") pod \"511ee639-1612-4e36-812e-70f7865291c1\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " Oct 31 17:28:24.768902 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.768800 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config\") pod \"511ee639-1612-4e36-812e-70f7865291c1\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " Oct 31 17:28:24.768902 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.768837 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn\") pod \"511ee639-1612-4e36-812e-70f7865291c1\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " Oct 31 17:28:24.768902 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.768870 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert\") pod \"511ee639-1612-4e36-812e-70f7865291c1\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " Oct 31 17:28:24.769287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.768955 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config\") pod \"511ee639-1612-4e36-812e-70f7865291c1\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " Oct 31 17:28:24.769287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.768997 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert\") pod \"511ee639-1612-4e36-812e-70f7865291c1\" (UID: \"511ee639-1612-4e36-812e-70f7865291c1\") " Oct 31 17:28:24.769287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.769074 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511ee639-1612-4e36-812e-70f7865291c1-tmp" pod "511ee639-1612-4e36-812e-70f7865291c1" (UID: "511ee639-1612-4e36-812e-70f7865291c1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:28:24.769287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.769197 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle" pod "511ee639-1612-4e36-812e-70f7865291c1" (UID: "511ee639-1612-4e36-812e-70f7865291c1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:24.769287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.769223 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config" pod "511ee639-1612-4e36-812e-70f7865291c1" (UID: "511ee639-1612-4e36-812e-70f7865291c1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:24.769525 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.769320 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca" pod "511ee639-1612-4e36-812e-70f7865291c1" (UID: "511ee639-1612-4e36-812e-70f7865291c1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:24.769525 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.769353 7618 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-trusted-ca-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:24.769525 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.769370 7618 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/511ee639-1612-4e36-812e-70f7865291c1-tmp\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:24.769525 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.769385 7618 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-console-config\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:24.769685 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.769663 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert" pod "511ee639-1612-4e36-812e-70f7865291c1" (UID: "511ee639-1612-4e36-812e-70f7865291c1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:24.771368 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.771347 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert" pod "511ee639-1612-4e36-812e-70f7865291c1" (UID: "511ee639-1612-4e36-812e-70f7865291c1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:24.771479 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.771371 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn" pod "511ee639-1612-4e36-812e-70f7865291c1" (UID: "511ee639-1612-4e36-812e-70f7865291c1"). InnerVolumeSpecName "kube-api-access-crxzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:28:24.771479 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.771403 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config" pod "511ee639-1612-4e36-812e-70f7865291c1" (UID: "511ee639-1612-4e36-812e-70f7865291c1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:24.870054 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.870019 7618 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:24.870054 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.870049 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crxzn\" (UniqueName: \"kubernetes.io/projected/511ee639-1612-4e36-812e-70f7865291c1-kube-api-access-crxzn\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:24.870054 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.870059 7618 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/511ee639-1612-4e36-812e-70f7865291c1-oauth-serving-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:24.870318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.870069 7618 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-oauth-config\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:24.870318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.870079 7618 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/511ee639-1612-4e36-812e-70f7865291c1-console-serving-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:24.949654 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.949573 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 17:28:24.949914 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.949875 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="alertmanager" containerID="cri-o://9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6" gracePeriod=120 Oct 31 17:28:24.950043 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.949937 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="kube-rbac-proxy-metric" containerID="cri-o://50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00" gracePeriod=120 Oct 31 17:28:24.950043 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.949988 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="kube-rbac-proxy-web" containerID="cri-o://fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578" gracePeriod=120 Oct 31 17:28:24.950325 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.950012 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="config-reloader" containerID="cri-o://5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5" gracePeriod=120 Oct 31 17:28:24.950325 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.950059 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="prom-label-proxy" containerID="cri-o://2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819" gracePeriod=120 Oct 31 17:28:24.950677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:24.950572 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="kube-rbac-proxy" containerID="cri-o://08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8" gracePeriod=120 Oct 31 17:28:25.224064 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.223990 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tdrsm" Oct 31 17:28:25.551337 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.551307 7618 generic.go:356] "Generic (PLEG): container finished" podID="176a27de-94c3-482d-9bcd-e335ba12400a" containerID="2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819" exitCode=0 Oct 31 17:28:25.551337 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.551331 7618 generic.go:356] "Generic (PLEG): container finished" podID="176a27de-94c3-482d-9bcd-e335ba12400a" containerID="08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8" exitCode=0 Oct 31 17:28:25.551813 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.551338 7618 generic.go:356] "Generic (PLEG): container finished" podID="176a27de-94c3-482d-9bcd-e335ba12400a" containerID="5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5" exitCode=0 Oct 31 17:28:25.551813 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.551359 7618 generic.go:356] "Generic (PLEG): container finished" podID="176a27de-94c3-482d-9bcd-e335ba12400a" containerID="9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6" exitCode=0 Oct 31 17:28:25.551813 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.551383 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerDied","Data":"2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819"} Oct 31 17:28:25.551813 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.551456 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerDied","Data":"08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8"} Oct 31 17:28:25.551813 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.551497 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerDied","Data":"5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5"} Oct 31 17:28:25.551813 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.551532 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerDied","Data":"9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6"} Oct 31 17:28:25.552720 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.552696 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f77577466-bjlt5" event={"ID":"511ee639-1612-4e36-812e-70f7865291c1","Type":"ContainerDied","Data":"b97cc610d95a14a810a8540c9c588346c8ab12383038645ca2f15bb2d9d88707"} Oct 31 17:28:25.552849 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.552741 7618 scope.go:122] "RemoveContainer" containerID="fbc6b961e24873f71116d543cefb5b67ceede670f46ec651fa891caa3a7065fe" Oct 31 17:28:25.552849 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.552742 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f77577466-bjlt5" Oct 31 17:28:25.595912 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.595879 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f77577466-bjlt5"] Oct 31 17:28:25.606158 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:25.606131 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f77577466-bjlt5"] Oct 31 17:28:26.225897 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.225871 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.370985 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.370913 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="511ee639-1612-4e36-812e-70f7865291c1" path="/var/lib/kubelet/pods/511ee639-1612-4e36-812e-70f7865291c1/volumes" Oct 31 17:28:26.384958 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.384934 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-config-out\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385086 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.384973 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-kube-api-access-q2mw5\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-kube-api-access-q2mw5\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385086 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385002 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-main-tls\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385086 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385037 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-web-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-web-config\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385084 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385239 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-trusted-ca-bundle\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385293 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-web\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385350 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-config-volume\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-config-volume\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385502 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385394 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-main-db\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385502 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385453 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-cluster-tls-config\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385502 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385493 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-tls-assets\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-tls-assets\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385644 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385535 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-metrics-client-ca\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385644 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385564 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy\") pod \"176a27de-94c3-482d-9bcd-e335ba12400a\" (UID: \"176a27de-94c3-482d-9bcd-e335ba12400a\") " Oct 31 17:28:26.385751 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385732 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-trusted-ca-bundle" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:26.385862 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.385842 7618 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.387784 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.387749 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-main-db" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:28:26.387930 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.387788 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-metrics-client-ca" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:26.388477 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.388432 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-metric" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:26.388663 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.388582 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-main-tls" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:26.388801 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.388773 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:26.388898 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.388807 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-config-out" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:28:26.388898 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.388874 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-kube-api-access-q2mw5" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "kube-api-access-q2mw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:28:26.389001 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.388901 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-web" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:26.389703 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.389682 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-tls-assets" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:28:26.389835 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.389816 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-config-volume" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:26.392770 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.392749 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-cluster-tls-config" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:26.399163 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.399133 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-web-config" pod "176a27de-94c3-482d-9bcd-e335ba12400a" (UID: "176a27de-94c3-482d-9bcd-e335ba12400a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:26.486694 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.486666 7618 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-web-config\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.486694 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.486696 7618 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.486870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.486708 7618 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.486870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.486719 7618 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-config-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.486870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.486729 7618 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-alertmanager-main-db\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.486870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.486738 7618 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-cluster-tls-config\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.486870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.486747 7618 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-tls-assets\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.486870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.486756 7618 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/176a27de-94c3-482d-9bcd-e335ba12400a-metrics-client-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.486870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.486765 7618 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.486870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.486773 7618 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/176a27de-94c3-482d-9bcd-e335ba12400a-config-out\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.486870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.486781 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2mw5\" (UniqueName: \"kubernetes.io/projected/176a27de-94c3-482d-9bcd-e335ba12400a-kube-api-access-q2mw5\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.486870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.486796 7618 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/176a27de-94c3-482d-9bcd-e335ba12400a-secret-alertmanager-main-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:26.559976 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.559944 7618 generic.go:356] "Generic (PLEG): container finished" podID="176a27de-94c3-482d-9bcd-e335ba12400a" containerID="50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00" exitCode=0 Oct 31 17:28:26.559976 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.559968 7618 generic.go:356] "Generic (PLEG): container finished" podID="176a27de-94c3-482d-9bcd-e335ba12400a" containerID="fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578" exitCode=0 Oct 31 17:28:26.560503 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.560025 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.560503 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.560024 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerDied","Data":"50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00"} Oct 31 17:28:26.560503 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.560106 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerDied","Data":"fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578"} Oct 31 17:28:26.560503 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.560152 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"176a27de-94c3-482d-9bcd-e335ba12400a","Type":"ContainerDied","Data":"65456a2e62791904d75b78dfdd341ade59d5c954ff3eb04009ea62d80d04030b"} Oct 31 17:28:26.560503 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.560168 7618 scope.go:122] "RemoveContainer" containerID="2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819" Oct 31 17:28:26.568541 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.568520 7618 scope.go:122] "RemoveContainer" containerID="50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00" Oct 31 17:28:26.575657 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.575641 7618 scope.go:122] "RemoveContainer" containerID="08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8" Oct 31 17:28:26.582306 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.582288 7618 scope.go:122] "RemoveContainer" containerID="fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578" Oct 31 17:28:26.589208 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.589193 7618 scope.go:122] "RemoveContainer" containerID="5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5" Oct 31 17:28:26.596320 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.596301 7618 scope.go:122] "RemoveContainer" containerID="9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6" Oct 31 17:28:26.605842 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.605821 7618 scope.go:122] "RemoveContainer" containerID="05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b" Oct 31 17:28:26.605971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.605884 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 17:28:26.613237 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.613217 7618 scope.go:122] "RemoveContainer" containerID="2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819" Oct 31 17:28:26.613555 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:26.613534 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819\": container with ID starting with 2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819 not found: ID does not exist" containerID="2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819" Oct 31 17:28:26.613630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.613564 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819"} err="failed to get container status \"2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819\": rpc error: code = NotFound desc = could not find container \"2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819\": container with ID starting with 2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819 not found: ID does not exist" Oct 31 17:28:26.613630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.613600 7618 scope.go:122] "RemoveContainer" containerID="50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00" Oct 31 17:28:26.613868 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:26.613841 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00\": container with ID starting with 50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00 not found: ID does not exist" containerID="50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00" Oct 31 17:28:26.613914 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.613877 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00"} err="failed to get container status \"50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00\": rpc error: code = NotFound desc = could not find container \"50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00\": container with ID starting with 50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00 not found: ID does not exist" Oct 31 17:28:26.613914 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.613894 7618 scope.go:122] "RemoveContainer" containerID="08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8" Oct 31 17:28:26.614148 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:26.614130 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8\": container with ID starting with 08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8 not found: ID does not exist" containerID="08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8" Oct 31 17:28:26.614196 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.614153 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8"} err="failed to get container status \"08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8\": rpc error: code = NotFound desc = could not find container \"08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8\": container with ID starting with 08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8 not found: ID does not exist" Oct 31 17:28:26.614196 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.614168 7618 scope.go:122] "RemoveContainer" containerID="fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578" Oct 31 17:28:26.614472 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:26.614453 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578\": container with ID starting with fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578 not found: ID does not exist" containerID="fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578" Oct 31 17:28:26.614540 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.614480 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578"} err="failed to get container status \"fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578\": rpc error: code = NotFound desc = could not find container \"fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578\": container with ID starting with fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578 not found: ID does not exist" Oct 31 17:28:26.614540 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.614509 7618 scope.go:122] "RemoveContainer" containerID="5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5" Oct 31 17:28:26.614770 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:26.614750 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5\": container with ID starting with 5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5 not found: ID does not exist" containerID="5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5" Oct 31 17:28:26.614821 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.614775 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5"} err="failed to get container status \"5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5\": rpc error: code = NotFound desc = could not find container \"5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5\": container with ID starting with 5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5 not found: ID does not exist" Oct 31 17:28:26.614821 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.614797 7618 scope.go:122] "RemoveContainer" containerID="9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6" Oct 31 17:28:26.615072 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:26.615053 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6\": container with ID starting with 9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6 not found: ID does not exist" containerID="9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6" Oct 31 17:28:26.615146 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.615077 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6"} err="failed to get container status \"9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6\": rpc error: code = NotFound desc = could not find container \"9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6\": container with ID starting with 9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6 not found: ID does not exist" Oct 31 17:28:26.615146 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.615097 7618 scope.go:122] "RemoveContainer" containerID="05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b" Oct 31 17:28:26.615433 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:26.615408 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b\": container with ID starting with 05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b not found: ID does not exist" containerID="05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b" Oct 31 17:28:26.615519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.615443 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b"} err="failed to get container status \"05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b\": rpc error: code = NotFound desc = could not find container \"05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b\": container with ID starting with 05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b not found: ID does not exist" Oct 31 17:28:26.615519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.615462 7618 scope.go:122] "RemoveContainer" containerID="2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819" Oct 31 17:28:26.615895 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.615860 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819"} err="failed to get container status \"2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819\": rpc error: code = NotFound desc = could not find container \"2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819\": container with ID starting with 2d4ec5576d0500b79414c9b0164311d56ac69e8b28d94a6a71c3cdf951055819 not found: ID does not exist" Oct 31 17:28:26.616006 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.615896 7618 scope.go:122] "RemoveContainer" containerID="50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00" Oct 31 17:28:26.616184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.616166 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00"} err="failed to get container status \"50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00\": rpc error: code = NotFound desc = could not find container \"50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00\": container with ID starting with 50e9f5e3c86ee35982ab752139f537de0e7efd886a3033f23032a271626ecf00 not found: ID does not exist" Oct 31 17:28:26.616239 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.616185 7618 scope.go:122] "RemoveContainer" containerID="08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8" Oct 31 17:28:26.616446 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.616427 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8"} err="failed to get container status \"08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8\": rpc error: code = NotFound desc = could not find container \"08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8\": container with ID starting with 08c317e5f24b468ce4ecf7ce388af4c1daba8777eab023d629426018ded3ded8 not found: ID does not exist" Oct 31 17:28:26.616519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.616447 7618 scope.go:122] "RemoveContainer" containerID="fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578" Oct 31 17:28:26.616646 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.616622 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 17:28:26.616710 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.616686 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578"} err="failed to get container status \"fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578\": rpc error: code = NotFound desc = could not find container \"fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578\": container with ID starting with fdb425b02642d605a601ca7be442dcc7a5e592abaf674237860c1878fcc35578 not found: ID does not exist" Oct 31 17:28:26.616710 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.616704 7618 scope.go:122] "RemoveContainer" containerID="5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5" Oct 31 17:28:26.616955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.616928 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5"} err="failed to get container status \"5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5\": rpc error: code = NotFound desc = could not find container \"5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5\": container with ID starting with 5cc4ec0f85dfdb9860ba163b4f85f2471f97ba50f260770be395c964a6e56dc5 not found: ID does not exist" Oct 31 17:28:26.617032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.616956 7618 scope.go:122] "RemoveContainer" containerID="9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6" Oct 31 17:28:26.617216 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.617196 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6"} err="failed to get container status \"9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6\": rpc error: code = NotFound desc = could not find container \"9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6\": container with ID starting with 9946442fa6273e0d56217603cc4cdbbba2a72df240600b23e8a256083450e5f6 not found: ID does not exist" Oct 31 17:28:26.617396 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.617216 7618 scope.go:122] "RemoveContainer" containerID="05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b" Oct 31 17:28:26.617546 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.617508 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b"} err="failed to get container status \"05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b\": rpc error: code = NotFound desc = could not find container \"05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b\": container with ID starting with 05af8b00d3860299dd446bc93e17a769a74a01fbb2fbb1eec4fc13f864a7728b not found: ID does not exist" Oct 31 17:28:26.635442 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635381 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 17:28:26.635666 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635655 7618 cpu_manager.go:414] "containerMap: removing container" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="kube-rbac-proxy-web" Oct 31 17:28:26.635706 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635668 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="kube-rbac-proxy-web" Oct 31 17:28:26.635706 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635685 7618 cpu_manager.go:414] "containerMap: removing container" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="alertmanager" Oct 31 17:28:26.635706 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635690 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="alertmanager" Oct 31 17:28:26.635706 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635697 7618 cpu_manager.go:414] "containerMap: removing container" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="config-reloader" Oct 31 17:28:26.635706 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635703 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="config-reloader" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635714 7618 cpu_manager.go:414] "containerMap: removing container" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="prom-label-proxy" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635719 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="prom-label-proxy" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635729 7618 cpu_manager.go:414] "containerMap: removing container" podUID="511ee639-1612-4e36-812e-70f7865291c1" containerName="console" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635734 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511ee639-1612-4e36-812e-70f7865291c1" containerName="console" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635743 7618 cpu_manager.go:414] "containerMap: removing container" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="kube-rbac-proxy-metric" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635747 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="kube-rbac-proxy-metric" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635756 7618 cpu_manager.go:414] "containerMap: removing container" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="kube-rbac-proxy" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635760 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="kube-rbac-proxy" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635767 7618 cpu_manager.go:414] "containerMap: removing container" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="init-config-reloader" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635771 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="init-config-reloader" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635818 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="511ee639-1612-4e36-812e-70f7865291c1" containerName="console" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635826 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="kube-rbac-proxy-metric" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635834 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="kube-rbac-proxy" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635842 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="init-config-reloader" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635850 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="kube-rbac-proxy-web" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635857 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="alertmanager" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635862 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="config-reloader" Oct 31 17:28:26.635863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.635868 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" containerName="prom-label-proxy" Oct 31 17:28:26.673391 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.673363 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 17:28:26.673533 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.673476 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.688086 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.688059 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 17:28:26.688217 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.688104 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 17:28:26.688339 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.688322 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 17:28:26.688339 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.688324 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 17:28:26.688440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.688344 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 17:28:26.688440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.688353 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 17:28:26.688539 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.688466 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 17:28:26.688539 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.688507 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hzhgj\"" Oct 31 17:28:26.688621 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.688607 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 17:28:26.702988 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.701071 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 17:28:26.789080 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789044 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-web-config\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.789246 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789087 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03ebb483-bb1a-471c-9363-0a8674cec02d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.789246 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789181 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03ebb483-bb1a-471c-9363-0a8674cec02d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.789246 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789209 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8r2q\" (UniqueName: \"kubernetes.io/projected/03ebb483-bb1a-471c-9363-0a8674cec02d-kube-api-access-x8r2q\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.789246 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789230 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03ebb483-bb1a-471c-9363-0a8674cec02d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.789397 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789355 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03ebb483-bb1a-471c-9363-0a8674cec02d-config-out\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.789433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789393 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.789433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789418 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.789499 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789478 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.789531 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789501 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.789531 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789520 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.789596 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789550 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03ebb483-bb1a-471c-9363-0a8674cec02d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.789628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.789619 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-config-volume\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.890851 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.890817 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891042 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.890862 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891042 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.890994 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891042 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891032 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891206 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891078 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03ebb483-bb1a-471c-9363-0a8674cec02d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891206 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891115 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-config-volume\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891206 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891152 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-web-config\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891206 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891194 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03ebb483-bb1a-471c-9363-0a8674cec02d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891414 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891219 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03ebb483-bb1a-471c-9363-0a8674cec02d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891414 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891235 7618 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 17:28:26.891604 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891243 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8r2q\" (UniqueName: \"kubernetes.io/projected/03ebb483-bb1a-471c-9363-0a8674cec02d-kube-api-access-x8r2q\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891650 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891610 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03ebb483-bb1a-471c-9363-0a8674cec02d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891681 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891664 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03ebb483-bb1a-471c-9363-0a8674cec02d-config-out\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891681 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891664 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03ebb483-bb1a-471c-9363-0a8674cec02d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.891741 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891699 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.892055 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.891986 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03ebb483-bb1a-471c-9363-0a8674cec02d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.892055 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.892031 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03ebb483-bb1a-471c-9363-0a8674cec02d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.894711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.894686 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.894859 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.894694 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.894978 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.894960 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03ebb483-bb1a-471c-9363-0a8674cec02d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.895076 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.895053 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.895233 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.895207 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-config-volume\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.895315 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.895222 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.895315 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.895276 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-web-config\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.895315 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.895300 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03ebb483-bb1a-471c-9363-0a8674cec02d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.895627 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.895610 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03ebb483-bb1a-471c-9363-0a8674cec02d-config-out\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:26.908689 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:26.908664 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8r2q\" (UniqueName: \"kubernetes.io/projected/03ebb483-bb1a-471c-9363-0a8674cec02d-kube-api-access-x8r2q\") pod \"alertmanager-main-0\" (UID: \"03ebb483-bb1a-471c-9363-0a8674cec02d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:27.104626 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:27.104589 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 17:28:27.458890 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:27.458793 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 17:28:27.463444 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:28:27.463415 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ebb483_bb1a_471c_9363_0a8674cec02d.slice/crio-806669e27c444a9fc99ea379ce1eadaa239a6abde3417e0413b44cc9eda71729 WatchSource:0}: Error finding container 806669e27c444a9fc99ea379ce1eadaa239a6abde3417e0413b44cc9eda71729: Status 404 returned error can't find the container with id 806669e27c444a9fc99ea379ce1eadaa239a6abde3417e0413b44cc9eda71729 Oct 31 17:28:27.567433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:27.567405 7618 generic.go:356] "Generic (PLEG): container finished" podID="03ebb483-bb1a-471c-9363-0a8674cec02d" containerID="5ac6662630ba3edff18c0a4cd59ba80e0435c5b81d9501012769902c5a20ff0b" exitCode=0 Oct 31 17:28:27.567901 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:27.567491 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03ebb483-bb1a-471c-9363-0a8674cec02d","Type":"ContainerDied","Data":"5ac6662630ba3edff18c0a4cd59ba80e0435c5b81d9501012769902c5a20ff0b"} Oct 31 17:28:27.567901 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:27.567553 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03ebb483-bb1a-471c-9363-0a8674cec02d","Type":"ContainerStarted","Data":"806669e27c444a9fc99ea379ce1eadaa239a6abde3417e0413b44cc9eda71729"} Oct 31 17:28:28.371813 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:28.371782 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176a27de-94c3-482d-9bcd-e335ba12400a" path="/var/lib/kubelet/pods/176a27de-94c3-482d-9bcd-e335ba12400a/volumes" Oct 31 17:28:28.574655 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:28.574618 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03ebb483-bb1a-471c-9363-0a8674cec02d","Type":"ContainerStarted","Data":"600849aa7aec4a8591811e53c3d7a2cb320e177349b5f588ebe8f4cf7d69653e"} Oct 31 17:28:28.575169 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:28.574696 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03ebb483-bb1a-471c-9363-0a8674cec02d","Type":"ContainerStarted","Data":"4785479eb0b4c7de2dd5396f8d598a3fb1c7ce0f1f366df7f91b958e0013f99a"} Oct 31 17:28:28.575169 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:28.574741 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03ebb483-bb1a-471c-9363-0a8674cec02d","Type":"ContainerStarted","Data":"ee8d397ccf387b5ef13ee163f94c1e9b68095922ebb487d4efeb763fcc84f2a0"} Oct 31 17:28:28.575169 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:28.574794 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03ebb483-bb1a-471c-9363-0a8674cec02d","Type":"ContainerStarted","Data":"51332a16a8f7e93f6f7a4b527f5a731acb27e2e9ad3df1ab6eb3081289f38a40"} Oct 31 17:28:28.575169 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:28.574842 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03ebb483-bb1a-471c-9363-0a8674cec02d","Type":"ContainerStarted","Data":"3d7d3175058b9e87bbe791a608b619dd14f4dc5a10202989f478d849369eb981"} Oct 31 17:28:28.575169 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:28.574880 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03ebb483-bb1a-471c-9363-0a8674cec02d","Type":"ContainerStarted","Data":"a8e5c6d2f30eff7a1ba21f83c35bfcc8567784b478215c82574d3362845ce6b0"} Oct 31 17:28:28.625963 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:28.625849 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.614245266 podStartE2EDuration="2.625830667s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.585401ms" isStatefulPod=false podCreationTimestamp="2026-10-31 17:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:28:28.624619619 +0000 UTC m=+30.372594086" watchObservedRunningTime="2026-10-31 17:28:28.625830667 +0000 UTC m=+30.373805134" Oct 31 17:28:29.536601 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.536565 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 17:28:29.536932 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.536881 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="prometheus" containerID="cri-o://55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b" gracePeriod=600 Oct 31 17:28:29.536932 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.536928 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="kube-rbac-proxy" containerID="cri-o://ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf" gracePeriod=600 Oct 31 17:28:29.537205 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.536929 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="config-reloader" containerID="cri-o://25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143" gracePeriod=600 Oct 31 17:28:29.537205 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.537077 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="thanos-sidecar" containerID="cri-o://91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363" gracePeriod=600 Oct 31 17:28:29.537205 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.537096 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="kube-rbac-proxy-thanos" containerID="cri-o://134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf" gracePeriod=600 Oct 31 17:28:29.537390 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.537352 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="kube-rbac-proxy-web" containerID="cri-o://09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419" gracePeriod=600 Oct 31 17:28:29.772784 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.772761 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:29.922381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922349 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-tls\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922392 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-serving-certs-ca-bundle\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922420 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-kubelet-serving-ca-bundle\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922444 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-thanos-prometheus-http-client-file\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922472 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922491 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-web-config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-web-config\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922522 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-0\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922557 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922588 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-metrics-client-ca\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922621 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-2\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922649 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-db\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922676 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-grpc-tls\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922725 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-trusted-ca-bundle\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922760 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-kube-api-access-zkkrp\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-kube-api-access-zkkrp\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922806 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-1\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922843 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-kube-rbac-proxy\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922879 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-config-out\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.922951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922937 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-config\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.923519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922947 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-kubelet-serving-ca-bundle" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:29.923519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922971 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-tls-assets\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-tls-assets\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.923519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.922975 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-serving-certs-ca-bundle" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:29.923519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.923001 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-metrics-client-certs\") pod \"9b556c44-090c-4e81-9d8d-2fac4026b0df\" (UID: \"9b556c44-090c-4e81-9d8d-2fac4026b0df\") " Oct 31 17:28:29.923519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.923113 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-2" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:29.923519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.923380 7618 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:29.923519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.923404 7618 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:29.923519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.923426 7618 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:29.923923 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.923897 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-db" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:28:29.924544 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.924512 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-0" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:29.925069 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.925042 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-thanos-prometheus-http-client-file" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:29.925734 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.925703 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-metrics-client-certs" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:29.925734 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.925721 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-trusted-ca-bundle" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:29.926052 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.926028 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-tls" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:29.926115 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.926066 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-1" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:29.926380 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.926358 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-grpc-tls" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:29.926456 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.926385 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-metrics-client-ca" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:28:29.926767 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.926698 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-thanos-sidecar-tls" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:29.927095 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.927068 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-kube-api-access-zkkrp" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "kube-api-access-zkkrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:28:29.927529 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.927500 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-kube-rbac-proxy-web" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:29.928068 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.928047 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-config-out" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:28:29.928204 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.928178 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-config" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:29.928309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.928279 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-kube-rbac-proxy" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:29.928443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.928425 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-tls-assets" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:28:29.936629 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:29.936604 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-web-config" pod "9b556c44-090c-4e81-9d8d-2fac4026b0df" (UID: "9b556c44-090c-4e81-9d8d-2fac4026b0df"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:28:30.024060 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024022 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkkrp\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-kube-api-access-zkkrp\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024060 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024054 7618 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024060 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024065 7618 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-kube-rbac-proxy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024075 7618 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-config-out\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024084 7618 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-config\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024092 7618 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b556c44-090c-4e81-9d8d-2fac4026b0df-tls-assets\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024100 7618 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-metrics-client-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024109 7618 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024118 7618 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024128 7618 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024136 7618 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-web-config\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024144 7618 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024153 7618 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024161 7618 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-configmap-metrics-client-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024170 7618 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-k8s-db\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024178 7618 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b556c44-090c-4e81-9d8d-2fac4026b0df-secret-grpc-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.024321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.024186 7618 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b556c44-090c-4e81-9d8d-2fac4026b0df-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:28:30.586799 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.586760 7618 generic.go:356] "Generic (PLEG): container finished" podID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerID="134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf" exitCode=0 Oct 31 17:28:30.586799 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.586786 7618 generic.go:356] "Generic (PLEG): container finished" podID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerID="ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf" exitCode=0 Oct 31 17:28:30.586799 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.586803 7618 generic.go:356] "Generic (PLEG): container finished" podID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerID="09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419" exitCode=0 Oct 31 17:28:30.587032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.586810 7618 generic.go:356] "Generic (PLEG): container finished" podID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerID="91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363" exitCode=0 Oct 31 17:28:30.587032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.586817 7618 generic.go:356] "Generic (PLEG): container finished" podID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerID="25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143" exitCode=0 Oct 31 17:28:30.587032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.586822 7618 generic.go:356] "Generic (PLEG): container finished" podID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerID="55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b" exitCode=0 Oct 31 17:28:30.587032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.586842 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerDied","Data":"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf"} Oct 31 17:28:30.587032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.586865 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.587032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.586938 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerDied","Data":"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf"} Oct 31 17:28:30.587032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.586965 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerDied","Data":"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419"} Oct 31 17:28:30.587032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.586990 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerDied","Data":"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363"} Oct 31 17:28:30.587032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.587003 7618 scope.go:122] "RemoveContainer" containerID="134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf" Oct 31 17:28:30.587032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.587017 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerDied","Data":"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143"} Oct 31 17:28:30.587438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.587041 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerDied","Data":"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b"} Oct 31 17:28:30.587438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.587064 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b556c44-090c-4e81-9d8d-2fac4026b0df","Type":"ContainerDied","Data":"854c83c8123f94442c3173892727408f7e91d92df7af64c41c7148f17cb8b9cf"} Oct 31 17:28:30.594537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.594517 7618 scope.go:122] "RemoveContainer" containerID="ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf" Oct 31 17:28:30.601299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.601278 7618 scope.go:122] "RemoveContainer" containerID="09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419" Oct 31 17:28:30.607644 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.607626 7618 scope.go:122] "RemoveContainer" containerID="91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363" Oct 31 17:28:30.614133 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.614117 7618 scope.go:122] "RemoveContainer" containerID="25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143" Oct 31 17:28:30.620629 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.620613 7618 scope.go:122] "RemoveContainer" containerID="55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b" Oct 31 17:28:30.628426 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.628355 7618 scope.go:122] "RemoveContainer" containerID="c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2" Oct 31 17:28:30.635702 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.635687 7618 scope.go:122] "RemoveContainer" containerID="134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf" Oct 31 17:28:30.635950 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:30.635934 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": container with ID starting with 134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf not found: ID does not exist" containerID="134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf" Oct 31 17:28:30.635991 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.635959 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf"} err="failed to get container status \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": rpc error: code = NotFound desc = could not find container \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": container with ID starting with 134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf not found: ID does not exist" Oct 31 17:28:30.635991 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.635975 7618 scope.go:122] "RemoveContainer" containerID="ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf" Oct 31 17:28:30.636187 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:30.636171 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": container with ID starting with ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf not found: ID does not exist" containerID="ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf" Oct 31 17:28:30.636222 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.636193 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf"} err="failed to get container status \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": rpc error: code = NotFound desc = could not find container \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": container with ID starting with ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf not found: ID does not exist" Oct 31 17:28:30.636222 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.636208 7618 scope.go:122] "RemoveContainer" containerID="09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419" Oct 31 17:28:30.636506 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:30.636490 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": container with ID starting with 09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419 not found: ID does not exist" containerID="09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419" Oct 31 17:28:30.636547 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.636511 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419"} err="failed to get container status \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": rpc error: code = NotFound desc = could not find container \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": container with ID starting with 09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419 not found: ID does not exist" Oct 31 17:28:30.636547 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.636523 7618 scope.go:122] "RemoveContainer" containerID="91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363" Oct 31 17:28:30.636716 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:30.636701 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": container with ID starting with 91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363 not found: ID does not exist" containerID="91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363" Oct 31 17:28:30.636756 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.636718 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363"} err="failed to get container status \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": rpc error: code = NotFound desc = could not find container \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": container with ID starting with 91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363 not found: ID does not exist" Oct 31 17:28:30.636756 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.636730 7618 scope.go:122] "RemoveContainer" containerID="25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143" Oct 31 17:28:30.636911 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:30.636896 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": container with ID starting with 25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143 not found: ID does not exist" containerID="25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143" Oct 31 17:28:30.636955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.636914 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143"} err="failed to get container status \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": rpc error: code = NotFound desc = could not find container \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": container with ID starting with 25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143 not found: ID does not exist" Oct 31 17:28:30.636955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.636926 7618 scope.go:122] "RemoveContainer" containerID="55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b" Oct 31 17:28:30.637117 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:30.637102 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": container with ID starting with 55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b not found: ID does not exist" containerID="55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b" Oct 31 17:28:30.637151 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.637119 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b"} err="failed to get container status \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": rpc error: code = NotFound desc = could not find container \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": container with ID starting with 55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b not found: ID does not exist" Oct 31 17:28:30.637151 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.637129 7618 scope.go:122] "RemoveContainer" containerID="c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2" Oct 31 17:28:30.637339 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:30.637325 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": container with ID starting with c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2 not found: ID does not exist" containerID="c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2" Oct 31 17:28:30.637389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.637341 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2"} err="failed to get container status \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": rpc error: code = NotFound desc = could not find container \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": container with ID starting with c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2 not found: ID does not exist" Oct 31 17:28:30.637389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.637352 7618 scope.go:122] "RemoveContainer" containerID="134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf" Oct 31 17:28:30.637554 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.637539 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf"} err="failed to get container status \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": rpc error: code = NotFound desc = could not find container \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": container with ID starting with 134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf not found: ID does not exist" Oct 31 17:28:30.637587 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.637554 7618 scope.go:122] "RemoveContainer" containerID="ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf" Oct 31 17:28:30.637742 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.637727 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf"} err="failed to get container status \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": rpc error: code = NotFound desc = could not find container \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": container with ID starting with ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf not found: ID does not exist" Oct 31 17:28:30.637784 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.637743 7618 scope.go:122] "RemoveContainer" containerID="09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419" Oct 31 17:28:30.637940 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.637926 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419"} err="failed to get container status \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": rpc error: code = NotFound desc = could not find container \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": container with ID starting with 09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419 not found: ID does not exist" Oct 31 17:28:30.637978 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.637939 7618 scope.go:122] "RemoveContainer" containerID="91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363" Oct 31 17:28:30.638115 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.638100 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363"} err="failed to get container status \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": rpc error: code = NotFound desc = could not find container \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": container with ID starting with 91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363 not found: ID does not exist" Oct 31 17:28:30.638162 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.638114 7618 scope.go:122] "RemoveContainer" containerID="25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143" Oct 31 17:28:30.638389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.638371 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143"} err="failed to get container status \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": rpc error: code = NotFound desc = could not find container \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": container with ID starting with 25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143 not found: ID does not exist" Oct 31 17:28:30.638389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.638388 7618 scope.go:122] "RemoveContainer" containerID="55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b" Oct 31 17:28:30.638622 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.638599 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b"} err="failed to get container status \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": rpc error: code = NotFound desc = could not find container \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": container with ID starting with 55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b not found: ID does not exist" Oct 31 17:28:30.638673 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.638623 7618 scope.go:122] "RemoveContainer" containerID="c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2" Oct 31 17:28:30.638827 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.638811 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2"} err="failed to get container status \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": rpc error: code = NotFound desc = could not find container \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": container with ID starting with c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2 not found: ID does not exist" Oct 31 17:28:30.638873 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.638827 7618 scope.go:122] "RemoveContainer" containerID="134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf" Oct 31 17:28:30.639023 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.639001 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf"} err="failed to get container status \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": rpc error: code = NotFound desc = could not find container \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": container with ID starting with 134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf not found: ID does not exist" Oct 31 17:28:30.639060 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.639023 7618 scope.go:122] "RemoveContainer" containerID="ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf" Oct 31 17:28:30.639208 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.639193 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf"} err="failed to get container status \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": rpc error: code = NotFound desc = could not find container \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": container with ID starting with ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf not found: ID does not exist" Oct 31 17:28:30.639265 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.639208 7618 scope.go:122] "RemoveContainer" containerID="09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419" Oct 31 17:28:30.639424 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.639409 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419"} err="failed to get container status \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": rpc error: code = NotFound desc = could not find container \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": container with ID starting with 09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419 not found: ID does not exist" Oct 31 17:28:30.639468 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.639423 7618 scope.go:122] "RemoveContainer" containerID="91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363" Oct 31 17:28:30.639610 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.639596 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363"} err="failed to get container status \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": rpc error: code = NotFound desc = could not find container \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": container with ID starting with 91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363 not found: ID does not exist" Oct 31 17:28:30.639649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.639610 7618 scope.go:122] "RemoveContainer" containerID="25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143" Oct 31 17:28:30.639789 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.639771 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143"} err="failed to get container status \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": rpc error: code = NotFound desc = could not find container \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": container with ID starting with 25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143 not found: ID does not exist" Oct 31 17:28:30.639789 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.639786 7618 scope.go:122] "RemoveContainer" containerID="55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b" Oct 31 17:28:30.639966 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.639950 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b"} err="failed to get container status \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": rpc error: code = NotFound desc = could not find container \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": container with ID starting with 55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b not found: ID does not exist" Oct 31 17:28:30.639966 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.639965 7618 scope.go:122] "RemoveContainer" containerID="c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2" Oct 31 17:28:30.640127 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.640111 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2"} err="failed to get container status \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": rpc error: code = NotFound desc = could not find container \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": container with ID starting with c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2 not found: ID does not exist" Oct 31 17:28:30.640165 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.640126 7618 scope.go:122] "RemoveContainer" containerID="134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf" Oct 31 17:28:30.640308 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.640293 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf"} err="failed to get container status \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": rpc error: code = NotFound desc = could not find container \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": container with ID starting with 134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf not found: ID does not exist" Oct 31 17:28:30.640357 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.640307 7618 scope.go:122] "RemoveContainer" containerID="ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf" Oct 31 17:28:30.640508 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.640493 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf"} err="failed to get container status \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": rpc error: code = NotFound desc = could not find container \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": container with ID starting with ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf not found: ID does not exist" Oct 31 17:28:30.640552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.640508 7618 scope.go:122] "RemoveContainer" containerID="09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419" Oct 31 17:28:30.640683 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.640668 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419"} err="failed to get container status \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": rpc error: code = NotFound desc = could not find container \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": container with ID starting with 09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419 not found: ID does not exist" Oct 31 17:28:30.640724 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.640682 7618 scope.go:122] "RemoveContainer" containerID="91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363" Oct 31 17:28:30.640856 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.640841 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363"} err="failed to get container status \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": rpc error: code = NotFound desc = could not find container \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": container with ID starting with 91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363 not found: ID does not exist" Oct 31 17:28:30.640893 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.640856 7618 scope.go:122] "RemoveContainer" containerID="25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143" Oct 31 17:28:30.641021 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.641006 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143"} err="failed to get container status \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": rpc error: code = NotFound desc = could not find container \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": container with ID starting with 25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143 not found: ID does not exist" Oct 31 17:28:30.641064 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.641020 7618 scope.go:122] "RemoveContainer" containerID="55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b" Oct 31 17:28:30.641212 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.641193 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b"} err="failed to get container status \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": rpc error: code = NotFound desc = could not find container \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": container with ID starting with 55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b not found: ID does not exist" Oct 31 17:28:30.641261 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.641214 7618 scope.go:122] "RemoveContainer" containerID="c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2" Oct 31 17:28:30.641417 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.641399 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2"} err="failed to get container status \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": rpc error: code = NotFound desc = could not find container \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": container with ID starting with c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2 not found: ID does not exist" Oct 31 17:28:30.641453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.641416 7618 scope.go:122] "RemoveContainer" containerID="134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf" Oct 31 17:28:30.641588 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.641573 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf"} err="failed to get container status \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": rpc error: code = NotFound desc = could not find container \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": container with ID starting with 134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf not found: ID does not exist" Oct 31 17:28:30.641625 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.641590 7618 scope.go:122] "RemoveContainer" containerID="ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf" Oct 31 17:28:30.641755 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.641739 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf"} err="failed to get container status \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": rpc error: code = NotFound desc = could not find container \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": container with ID starting with ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf not found: ID does not exist" Oct 31 17:28:30.641793 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.641755 7618 scope.go:122] "RemoveContainer" containerID="09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419" Oct 31 17:28:30.641934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.641919 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419"} err="failed to get container status \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": rpc error: code = NotFound desc = could not find container \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": container with ID starting with 09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419 not found: ID does not exist" Oct 31 17:28:30.641934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.641933 7618 scope.go:122] "RemoveContainer" containerID="91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363" Oct 31 17:28:30.642105 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.642090 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363"} err="failed to get container status \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": rpc error: code = NotFound desc = could not find container \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": container with ID starting with 91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363 not found: ID does not exist" Oct 31 17:28:30.642105 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.642104 7618 scope.go:122] "RemoveContainer" containerID="25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143" Oct 31 17:28:30.642290 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.642273 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143"} err="failed to get container status \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": rpc error: code = NotFound desc = could not find container \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": container with ID starting with 25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143 not found: ID does not exist" Oct 31 17:28:30.642290 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.642289 7618 scope.go:122] "RemoveContainer" containerID="55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b" Oct 31 17:28:30.642482 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.642465 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b"} err="failed to get container status \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": rpc error: code = NotFound desc = could not find container \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": container with ID starting with 55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b not found: ID does not exist" Oct 31 17:28:30.642482 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.642481 7618 scope.go:122] "RemoveContainer" containerID="c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2" Oct 31 17:28:30.642655 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.642641 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2"} err="failed to get container status \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": rpc error: code = NotFound desc = could not find container \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": container with ID starting with c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2 not found: ID does not exist" Oct 31 17:28:30.642696 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.642655 7618 scope.go:122] "RemoveContainer" containerID="134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf" Oct 31 17:28:30.642841 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.642825 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf"} err="failed to get container status \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": rpc error: code = NotFound desc = could not find container \"134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf\": container with ID starting with 134171762a146057892507747421afb529b4d91082d8203d031a6cccd97744cf not found: ID does not exist" Oct 31 17:28:30.642841 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.642840 7618 scope.go:122] "RemoveContainer" containerID="ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf" Oct 31 17:28:30.643017 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.643002 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf"} err="failed to get container status \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": rpc error: code = NotFound desc = could not find container \"ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf\": container with ID starting with ec94be8b74ae471b96cc2fc9402f50ff24cc3fc56bd7fe736f21bfc9fb1b65cf not found: ID does not exist" Oct 31 17:28:30.643059 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.643016 7618 scope.go:122] "RemoveContainer" containerID="09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419" Oct 31 17:28:30.643204 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.643189 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419"} err="failed to get container status \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": rpc error: code = NotFound desc = could not find container \"09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419\": container with ID starting with 09d1eb18f2b12499dff967029159b75f5d016eb98c614f7486904eb032e1c419 not found: ID does not exist" Oct 31 17:28:30.643204 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.643203 7618 scope.go:122] "RemoveContainer" containerID="91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363" Oct 31 17:28:30.643397 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.643382 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363"} err="failed to get container status \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": rpc error: code = NotFound desc = could not find container \"91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363\": container with ID starting with 91be36ff4253750fa790f64bb81ba0c22941381c21f51547976041603bfd8363 not found: ID does not exist" Oct 31 17:28:30.643446 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.643397 7618 scope.go:122] "RemoveContainer" containerID="25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143" Oct 31 17:28:30.643567 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.643552 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143"} err="failed to get container status \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": rpc error: code = NotFound desc = could not find container \"25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143\": container with ID starting with 25977db0d03351f674ccfe659fd4ddc0a7126a27392ac3311c20ec25f989c143 not found: ID does not exist" Oct 31 17:28:30.643608 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.643566 7618 scope.go:122] "RemoveContainer" containerID="55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b" Oct 31 17:28:30.643807 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.643791 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b"} err="failed to get container status \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": rpc error: code = NotFound desc = could not find container \"55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b\": container with ID starting with 55820c650426a774811bc42e663c49fbf42b1bd65ed22339fb3459d944f0047b not found: ID does not exist" Oct 31 17:28:30.643844 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.643807 7618 scope.go:122] "RemoveContainer" containerID="c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2" Oct 31 17:28:30.644054 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.644035 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2"} err="failed to get container status \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": rpc error: code = NotFound desc = could not find container \"c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2\": container with ID starting with c39c50f7894719718a794b030107e5fe391d18b23457f546d9d8f308421163b2 not found: ID does not exist" Oct 31 17:28:30.644842 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.644825 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 17:28:30.655077 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.655058 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 17:28:30.676587 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676558 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 17:28:30.676839 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676828 7618 cpu_manager.go:414] "containerMap: removing container" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="prometheus" Oct 31 17:28:30.676880 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676840 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="prometheus" Oct 31 17:28:30.676880 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676857 7618 cpu_manager.go:414] "containerMap: removing container" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="config-reloader" Oct 31 17:28:30.676880 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676862 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="config-reloader" Oct 31 17:28:30.676880 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676874 7618 cpu_manager.go:414] "containerMap: removing container" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="kube-rbac-proxy" Oct 31 17:28:30.676880 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676879 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="kube-rbac-proxy" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676888 7618 cpu_manager.go:414] "containerMap: removing container" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="thanos-sidecar" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676893 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="thanos-sidecar" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676902 7618 cpu_manager.go:414] "containerMap: removing container" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="kube-rbac-proxy-thanos" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676907 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="kube-rbac-proxy-thanos" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676913 7618 cpu_manager.go:414] "containerMap: removing container" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="kube-rbac-proxy-web" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676918 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="kube-rbac-proxy-web" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676930 7618 cpu_manager.go:414] "containerMap: removing container" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="init-config-reloader" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676934 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="init-config-reloader" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676979 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="kube-rbac-proxy-thanos" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676986 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="kube-rbac-proxy-web" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.676993 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="init-config-reloader" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.677000 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="prometheus" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.677006 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="config-reloader" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.677012 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="kube-rbac-proxy" Oct 31 17:28:30.677019 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.677017 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" containerName="thanos-sidecar" Oct 31 17:28:30.681346 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.681327 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.694915 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.694892 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 17:28:30.694915 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.694903 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 17:28:30.695092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.695004 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 17:28:30.695092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.695049 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 17:28:30.695092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.695064 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 17:28:30.695311 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.695200 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 17:28:30.695311 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.695242 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 17:28:30.695418 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.695331 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 17:28:30.695418 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.695405 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ccmlejc9fapc\"" Oct 31 17:28:30.695516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.695458 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 17:28:30.695516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.695485 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 17:28:30.704911 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.704894 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 17:28:30.705030 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.704950 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6jpgp\"" Oct 31 17:28:30.705030 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.704981 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 17:28:30.706104 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.706087 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 17:28:30.706506 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.706490 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 17:28:30.709357 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.709247 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 17:28:30.831441 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831408 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831441 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831447 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831506 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831533 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831609 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3caff88e-0edc-4fac-9a1c-1387610e526f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831637 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831660 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831677 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831743 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3caff88e-0edc-4fac-9a1c-1387610e526f-config-out\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831780 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831815 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831837 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831867 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831886 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-config\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831920 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.831957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831938 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.832478 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.831971 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.832478 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.832039 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsf68\" (UniqueName: \"kubernetes.io/projected/3caff88e-0edc-4fac-9a1c-1387610e526f-kube-api-access-nsf68\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.832478 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.832058 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-web-config\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.832478 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.832139 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.932866 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.932827 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933034 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.932915 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-config\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933034 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.932960 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933034 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.932980 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933034 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933002 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933034 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933025 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsf68\" (UniqueName: \"kubernetes.io/projected/3caff88e-0edc-4fac-9a1c-1387610e526f-kube-api-access-nsf68\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933044 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-web-config\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933074 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933100 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933140 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933168 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933192 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933236 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3caff88e-0edc-4fac-9a1c-1387610e526f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933286 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933319 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933792 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933341 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933792 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933371 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3caff88e-0edc-4fac-9a1c-1387610e526f-config-out\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933792 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933394 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933792 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933423 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933792 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933453 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.933792 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933749 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.934108 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.933918 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.934168 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.934126 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.935628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.934685 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.935628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.934920 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.935628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.935334 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.935628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.935554 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.936826 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.936802 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-web-config\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.936981 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.936956 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3caff88e-0edc-4fac-9a1c-1387610e526f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.937342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.937315 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3caff88e-0edc-4fac-9a1c-1387610e526f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.937436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.937399 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.937502 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.937468 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.937950 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.937926 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-config\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.938586 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.938552 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.938586 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.938569 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.938715 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.938683 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.938887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.938866 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.939276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.939240 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3caff88e-0edc-4fac-9a1c-1387610e526f-config-out\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.939379 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.939359 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3caff88e-0edc-4fac-9a1c-1387610e526f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:30.950350 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:30.950329 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsf68\" (UniqueName: \"kubernetes.io/projected/3caff88e-0edc-4fac-9a1c-1387610e526f-kube-api-access-nsf68\") pod \"prometheus-k8s-0\" (UID: \"3caff88e-0edc-4fac-9a1c-1387610e526f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:31.003347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:31.003311 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:31.324558 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:31.324468 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 17:28:31.329181 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:28:31.329152 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3caff88e_0edc_4fac_9a1c_1387610e526f.slice/crio-ec5ea49e87e93503c076e72e83e121740fb6eddfa8b5e626feb187ec7284fb78 WatchSource:0}: Error finding container ec5ea49e87e93503c076e72e83e121740fb6eddfa8b5e626feb187ec7284fb78: Status 404 returned error can't find the container with id ec5ea49e87e93503c076e72e83e121740fb6eddfa8b5e626feb187ec7284fb78 Oct 31 17:28:31.592202 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:31.592110 7618 generic.go:356] "Generic (PLEG): container finished" podID="3caff88e-0edc-4fac-9a1c-1387610e526f" containerID="8d9e21ade1dc7b8590de56396ed98fcf139adad2e0579287f4294b474b013788" exitCode=0 Oct 31 17:28:31.592370 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:31.592201 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3caff88e-0edc-4fac-9a1c-1387610e526f","Type":"ContainerDied","Data":"8d9e21ade1dc7b8590de56396ed98fcf139adad2e0579287f4294b474b013788"} Oct 31 17:28:31.592370 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:31.592301 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3caff88e-0edc-4fac-9a1c-1387610e526f","Type":"ContainerStarted","Data":"ec5ea49e87e93503c076e72e83e121740fb6eddfa8b5e626feb187ec7284fb78"} Oct 31 17:28:31.719303 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:31.719279 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cd9s8" Oct 31 17:28:32.371969 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:32.371934 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b556c44-090c-4e81-9d8d-2fac4026b0df" path="/var/lib/kubelet/pods/9b556c44-090c-4e81-9d8d-2fac4026b0df/volumes" Oct 31 17:28:32.600809 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:32.600777 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3caff88e-0edc-4fac-9a1c-1387610e526f","Type":"ContainerStarted","Data":"2b964c532bf28c2df915e8359b9f307e7638320e9d80c2d04c92f521fdaaf097"} Oct 31 17:28:32.600994 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:32.600843 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3caff88e-0edc-4fac-9a1c-1387610e526f","Type":"ContainerStarted","Data":"f80b4d04a123f4420f5f776243d5f0f5e0c09022e21d279e58767060bdd14352"} Oct 31 17:28:32.600994 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:32.600874 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3caff88e-0edc-4fac-9a1c-1387610e526f","Type":"ContainerStarted","Data":"cfd01f11062365f1f4e2eb85297a0c645acf3d91453124402808a06e3b25fb4f"} Oct 31 17:28:32.600994 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:32.600899 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3caff88e-0edc-4fac-9a1c-1387610e526f","Type":"ContainerStarted","Data":"22c0bc16908112386971ecb3b737532da398bdb880978c92027546caf24bf119"} Oct 31 17:28:32.600994 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:32.600927 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3caff88e-0edc-4fac-9a1c-1387610e526f","Type":"ContainerStarted","Data":"cb63d1517d36bec15b9de66ac0d338fed84f99be9c00afb86af4f7fdb8354709"} Oct 31 17:28:32.600994 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:32.600957 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3caff88e-0edc-4fac-9a1c-1387610e526f","Type":"ContainerStarted","Data":"0c78e86e961fc35c7c454598676f6d3515cac32006086e01d55c31bcde7d3e2f"} Oct 31 17:28:32.654944 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:32.654894 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.635961661 podStartE2EDuration="2.654881495s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.919834ms" isStatefulPod=false podCreationTimestamp="2026-10-31 17:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:28:32.653858795 +0000 UTC m=+34.401833261" watchObservedRunningTime="2026-10-31 17:28:32.654881495 +0000 UTC m=+34.402855956" Oct 31 17:28:36.003814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:28:36.003763 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:28:38.571190 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:28:38.571153 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:29:08.570989 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:29:08.570953 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:29:09.880486 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:29:09.880446 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:29:09.896036 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:29:09.896009 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:29:10.756026 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:29:10.756000 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 17:29:38.571168 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:29:38.571127 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:30:08.574555 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:30:08.574516 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:30:12.402111 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.402073 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7"] Oct 31 17:30:12.405069 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.405043 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:12.418894 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.418869 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7\" (UID: \"e6007407-99e6-4735-8ee5-b20a10ed0205\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:12.419013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.418934 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4s59\" (UniqueName: \"kubernetes.io/projected/e6007407-99e6-4735-8ee5-b20a10ed0205-kube-api-access-j4s59\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7\" (UID: \"e6007407-99e6-4735-8ee5-b20a10ed0205\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:12.419013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.418999 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7\" (UID: \"e6007407-99e6-4735-8ee5-b20a10ed0205\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:12.427896 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.427865 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 17:30:12.428054 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.427971 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-rd8lp\"" Oct 31 17:30:12.428115 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.428080 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 17:30:12.428704 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.428680 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7"] Oct 31 17:30:12.519900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.519862 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4s59\" (UniqueName: \"kubernetes.io/projected/e6007407-99e6-4735-8ee5-b20a10ed0205-kube-api-access-j4s59\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7\" (UID: \"e6007407-99e6-4735-8ee5-b20a10ed0205\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:12.520107 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.519917 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7\" (UID: \"e6007407-99e6-4735-8ee5-b20a10ed0205\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:12.520107 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.519945 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7\" (UID: \"e6007407-99e6-4735-8ee5-b20a10ed0205\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:12.520335 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.520318 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7\" (UID: \"e6007407-99e6-4735-8ee5-b20a10ed0205\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:12.520429 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.520409 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7\" (UID: \"e6007407-99e6-4735-8ee5-b20a10ed0205\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:12.537092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.537059 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4s59\" (UniqueName: \"kubernetes.io/projected/e6007407-99e6-4735-8ee5-b20a10ed0205-kube-api-access-j4s59\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7\" (UID: \"e6007407-99e6-4735-8ee5-b20a10ed0205\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:12.718595 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:12.718507 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:13.008386 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:13.008296 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7"] Oct 31 17:30:13.013767 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:30:13.013730 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6007407_99e6_4735_8ee5_b20a10ed0205.slice/crio-8f07d1f2d4184a29a280e3d3488ec5654f30a3d9331b8e8120528edf824ccc6b WatchSource:0}: Error finding container 8f07d1f2d4184a29a280e3d3488ec5654f30a3d9331b8e8120528edf824ccc6b: Status 404 returned error can't find the container with id 8f07d1f2d4184a29a280e3d3488ec5654f30a3d9331b8e8120528edf824ccc6b Oct 31 17:30:13.015963 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:13.015945 7618 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 17:30:13.967704 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:13.967666 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" event={"ID":"e6007407-99e6-4735-8ee5-b20a10ed0205","Type":"ContainerStarted","Data":"8f07d1f2d4184a29a280e3d3488ec5654f30a3d9331b8e8120528edf824ccc6b"} Oct 31 17:30:14.387031 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.386946 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp"] Oct 31 17:30:14.389817 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.389794 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:14.413288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.413241 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-dprkl\"" Oct 31 17:30:14.414272 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.414225 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp"] Oct 31 17:30:14.414846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.414816 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d"] Oct 31 17:30:14.417550 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.417529 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:14.435324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.435292 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d\" (UID: \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:14.435574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.435541 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsq2m\" (UniqueName: \"kubernetes.io/projected/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-kube-api-access-jsq2m\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d\" (UID: \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:14.435574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.435607 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp\" (UID: \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:14.435574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.435683 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp\" (UID: \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:14.435574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.435739 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d\" (UID: \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:14.435574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.435787 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6k5z\" (UniqueName: \"kubernetes.io/projected/f9b2a7d1-4ee9-429d-8665-3bce025cf287-kube-api-access-m6k5z\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp\" (UID: \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:14.435574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.436230 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d"] Oct 31 17:30:14.503292 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.503244 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-7wx69\"" Oct 31 17:30:14.536484 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.536448 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp\" (UID: \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:14.536685 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.536501 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp\" (UID: \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:14.536757 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.536687 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d\" (UID: \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:14.536757 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.536743 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6k5z\" (UniqueName: \"kubernetes.io/projected/f9b2a7d1-4ee9-429d-8665-3bce025cf287-kube-api-access-m6k5z\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp\" (UID: \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:14.536873 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.536765 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d\" (UID: \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:14.536873 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.536795 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsq2m\" (UniqueName: \"kubernetes.io/projected/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-kube-api-access-jsq2m\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d\" (UID: \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:14.536976 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.536924 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp\" (UID: \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:14.537030 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.536982 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp\" (UID: \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:14.537304 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.537281 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d\" (UID: \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:14.537437 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.537358 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d\" (UID: \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:14.554669 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.554642 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsq2m\" (UniqueName: \"kubernetes.io/projected/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-kube-api-access-jsq2m\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d\" (UID: \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:14.554812 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.554755 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6k5z\" (UniqueName: \"kubernetes.io/projected/f9b2a7d1-4ee9-429d-8665-3bce025cf287-kube-api-access-m6k5z\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp\" (UID: \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:14.706307 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.706242 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:14.734341 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:14.734307 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:15.008686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:15.008578 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp"] Oct 31 17:30:15.016473 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:30:15.016437 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b2a7d1_4ee9_429d_8665_3bce025cf287.slice/crio-d473d43382abcdf1c89e17dc4f9c9ac794bf9234fb90b6fbb78acb6913b89bd4 WatchSource:0}: Error finding container d473d43382abcdf1c89e17dc4f9c9ac794bf9234fb90b6fbb78acb6913b89bd4: Status 404 returned error can't find the container with id d473d43382abcdf1c89e17dc4f9c9ac794bf9234fb90b6fbb78acb6913b89bd4 Oct 31 17:30:15.042693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:15.042657 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d"] Oct 31 17:30:15.047926 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:30:15.047900 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5672ab6_0efe_46ba_ac22_b8ca51b7de16.slice/crio-52679a1bc9fbccf31e7c6cd09335012ce43cff58cc3c7be3431126efe4157a62 WatchSource:0}: Error finding container 52679a1bc9fbccf31e7c6cd09335012ce43cff58cc3c7be3431126efe4157a62: Status 404 returned error can't find the container with id 52679a1bc9fbccf31e7c6cd09335012ce43cff58cc3c7be3431126efe4157a62 Oct 31 17:30:15.983524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:15.979971 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" event={"ID":"f5672ab6-0efe-46ba-ac22-b8ca51b7de16","Type":"ContainerStarted","Data":"52679a1bc9fbccf31e7c6cd09335012ce43cff58cc3c7be3431126efe4157a62"} Oct 31 17:30:15.983524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:15.981622 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" event={"ID":"f9b2a7d1-4ee9-429d-8665-3bce025cf287","Type":"ContainerStarted","Data":"d473d43382abcdf1c89e17dc4f9c9ac794bf9234fb90b6fbb78acb6913b89bd4"} Oct 31 17:30:20.005947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:20.005861 7618 generic.go:356] "Generic (PLEG): container finished" podID="e6007407-99e6-4735-8ee5-b20a10ed0205" containerID="3719d3d9bb43e0c55abcb3ea9b8a62b173eb037c91e6aaeb02b12a9b9ecb20fd" exitCode=0 Oct 31 17:30:20.005947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:20.005903 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" event={"ID":"e6007407-99e6-4735-8ee5-b20a10ed0205","Type":"ContainerDied","Data":"3719d3d9bb43e0c55abcb3ea9b8a62b173eb037c91e6aaeb02b12a9b9ecb20fd"} Oct 31 17:30:22.016500 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:22.016468 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" event={"ID":"f9b2a7d1-4ee9-429d-8665-3bce025cf287","Type":"ContainerStarted","Data":"fd90e4552ed7a34d74939d5eadae9ffd8171ca9a1a8f638bfd6185d22daef656"} Oct 31 17:30:22.018153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:22.018122 7618 generic.go:356] "Generic (PLEG): container finished" podID="f5672ab6-0efe-46ba-ac22-b8ca51b7de16" containerID="06a7ea9773bee55f268b4edab15bd9fd37895cce7a2ba1a6ec6034b78c41f318" exitCode=0 Oct 31 17:30:22.018322 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:22.018176 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" event={"ID":"f5672ab6-0efe-46ba-ac22-b8ca51b7de16","Type":"ContainerDied","Data":"06a7ea9773bee55f268b4edab15bd9fd37895cce7a2ba1a6ec6034b78c41f318"} Oct 31 17:30:23.023337 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:23.023308 7618 generic.go:356] "Generic (PLEG): container finished" podID="f9b2a7d1-4ee9-429d-8665-3bce025cf287" containerID="fd90e4552ed7a34d74939d5eadae9ffd8171ca9a1a8f638bfd6185d22daef656" exitCode=0 Oct 31 17:30:23.023689 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:23.023394 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" event={"ID":"f9b2a7d1-4ee9-429d-8665-3bce025cf287","Type":"ContainerDied","Data":"fd90e4552ed7a34d74939d5eadae9ffd8171ca9a1a8f638bfd6185d22daef656"} Oct 31 17:30:23.025237 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:23.025214 7618 generic.go:356] "Generic (PLEG): container finished" podID="e6007407-99e6-4735-8ee5-b20a10ed0205" containerID="e127a0679255db0ca471c4e8b18640c4d381cac896358b667e00c05f43b371c3" exitCode=0 Oct 31 17:30:23.025352 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:23.025285 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" event={"ID":"e6007407-99e6-4735-8ee5-b20a10ed0205","Type":"ContainerDied","Data":"e127a0679255db0ca471c4e8b18640c4d381cac896358b667e00c05f43b371c3"} Oct 31 17:30:24.031517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:24.031483 7618 generic.go:356] "Generic (PLEG): container finished" podID="f5672ab6-0efe-46ba-ac22-b8ca51b7de16" containerID="f265e8ecc208a6f2df91b9ce40e7425ce42f8ca6185c544f940ea25e98ecfa07" exitCode=0 Oct 31 17:30:24.031927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:24.031567 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" event={"ID":"f5672ab6-0efe-46ba-ac22-b8ca51b7de16","Type":"ContainerDied","Data":"f265e8ecc208a6f2df91b9ce40e7425ce42f8ca6185c544f940ea25e98ecfa07"} Oct 31 17:30:25.038527 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:25.038482 7618 generic.go:356] "Generic (PLEG): container finished" podID="f9b2a7d1-4ee9-429d-8665-3bce025cf287" containerID="f5aa0f05d260f0deb9172b6549cba82ce3b201b9fe92ed47a4bf8937f2a4ca12" exitCode=0 Oct 31 17:30:25.039000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:25.038538 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" event={"ID":"f9b2a7d1-4ee9-429d-8665-3bce025cf287","Type":"ContainerDied","Data":"f5aa0f05d260f0deb9172b6549cba82ce3b201b9fe92ed47a4bf8937f2a4ca12"} Oct 31 17:30:31.072846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:31.072813 7618 generic.go:356] "Generic (PLEG): container finished" podID="e6007407-99e6-4735-8ee5-b20a10ed0205" containerID="4e04259367b225c44c91a8358a59166437c4bb4b5d61949930ef4288ce8a4961" exitCode=0 Oct 31 17:30:31.073328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:31.072884 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" event={"ID":"e6007407-99e6-4735-8ee5-b20a10ed0205","Type":"ContainerDied","Data":"4e04259367b225c44c91a8358a59166437c4bb4b5d61949930ef4288ce8a4961"} Oct 31 17:30:32.079420 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:32.079382 7618 generic.go:356] "Generic (PLEG): container finished" podID="f5672ab6-0efe-46ba-ac22-b8ca51b7de16" containerID="40faa2377cc9a427da6b2f47d0d934cc73f77df462c68ce67f2447d237d6d25e" exitCode=0 Oct 31 17:30:32.079895 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:32.079467 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" event={"ID":"f5672ab6-0efe-46ba-ac22-b8ca51b7de16","Type":"ContainerDied","Data":"40faa2377cc9a427da6b2f47d0d934cc73f77df462c68ce67f2447d237d6d25e"} Oct 31 17:30:32.222885 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:32.222864 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:32.305534 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:32.305503 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6007407-99e6-4735-8ee5-b20a10ed0205-kube-api-access-j4s59\" (UniqueName: \"kubernetes.io/projected/e6007407-99e6-4735-8ee5-b20a10ed0205-kube-api-access-j4s59\") pod \"e6007407-99e6-4735-8ee5-b20a10ed0205\" (UID: \"e6007407-99e6-4735-8ee5-b20a10ed0205\") " Oct 31 17:30:32.305736 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:32.305571 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-util\" (UniqueName: \"kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-util\") pod \"e6007407-99e6-4735-8ee5-b20a10ed0205\" (UID: \"e6007407-99e6-4735-8ee5-b20a10ed0205\") " Oct 31 17:30:32.305736 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:32.305600 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-bundle\") pod \"e6007407-99e6-4735-8ee5-b20a10ed0205\" (UID: \"e6007407-99e6-4735-8ee5-b20a10ed0205\") " Oct 31 17:30:32.306045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:32.306019 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-bundle" pod "e6007407-99e6-4735-8ee5-b20a10ed0205" (UID: "e6007407-99e6-4735-8ee5-b20a10ed0205"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:30:32.308081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:32.308054 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6007407-99e6-4735-8ee5-b20a10ed0205-kube-api-access-j4s59" pod "e6007407-99e6-4735-8ee5-b20a10ed0205" (UID: "e6007407-99e6-4735-8ee5-b20a10ed0205"). InnerVolumeSpecName "kube-api-access-j4s59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:30:32.310092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:32.310069 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-util" pod "e6007407-99e6-4735-8ee5-b20a10ed0205" (UID: "e6007407-99e6-4735-8ee5-b20a10ed0205"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:30:32.406781 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:32.406753 7618 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-util\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:30:32.406781 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:32.406783 7618 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6007407-99e6-4735-8ee5-b20a10ed0205-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:30:32.406935 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:32.406796 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4s59\" (UniqueName: \"kubernetes.io/projected/e6007407-99e6-4735-8ee5-b20a10ed0205-kube-api-access-j4s59\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:30:33.085558 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.085518 7618 generic.go:356] "Generic (PLEG): container finished" podID="f9b2a7d1-4ee9-429d-8665-3bce025cf287" containerID="8ac6c96a235aad5e063e92bc323ca45b84d917bb71c1aa5b4352637dec17ba95" exitCode=0 Oct 31 17:30:33.086024 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.085607 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" event={"ID":"f9b2a7d1-4ee9-429d-8665-3bce025cf287","Type":"ContainerDied","Data":"8ac6c96a235aad5e063e92bc323ca45b84d917bb71c1aa5b4352637dec17ba95"} Oct 31 17:30:33.087370 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.087342 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" event={"ID":"e6007407-99e6-4735-8ee5-b20a10ed0205","Type":"ContainerDied","Data":"8f07d1f2d4184a29a280e3d3488ec5654f30a3d9331b8e8120528edf824ccc6b"} Oct 31 17:30:33.087488 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.087371 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644t2rp7" Oct 31 17:30:33.087488 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.087386 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f07d1f2d4184a29a280e3d3488ec5654f30a3d9331b8e8120528edf824ccc6b" Oct 31 17:30:33.208947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.208921 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:33.313634 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.313598 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-util\" (UniqueName: \"kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-util\") pod \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\" (UID: \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\") " Oct 31 17:30:33.313811 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.313667 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-kube-api-access-jsq2m\" (UniqueName: \"kubernetes.io/projected/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-kube-api-access-jsq2m\") pod \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\" (UID: \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\") " Oct 31 17:30:33.313811 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.313689 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-bundle\") pod \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\" (UID: \"f5672ab6-0efe-46ba-ac22-b8ca51b7de16\") " Oct 31 17:30:33.314224 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.314199 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-bundle" pod "f5672ab6-0efe-46ba-ac22-b8ca51b7de16" (UID: "f5672ab6-0efe-46ba-ac22-b8ca51b7de16"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:30:33.315752 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.315727 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-kube-api-access-jsq2m" pod "f5672ab6-0efe-46ba-ac22-b8ca51b7de16" (UID: "f5672ab6-0efe-46ba-ac22-b8ca51b7de16"). InnerVolumeSpecName "kube-api-access-jsq2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:30:33.319124 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.319100 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-util" pod "f5672ab6-0efe-46ba-ac22-b8ca51b7de16" (UID: "f5672ab6-0efe-46ba-ac22-b8ca51b7de16"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:30:33.414316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.414282 7618 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-util\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:30:33.414316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.414310 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsq2m\" (UniqueName: \"kubernetes.io/projected/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-kube-api-access-jsq2m\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:30:33.414316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:33.414320 7618 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5672ab6-0efe-46ba-ac22-b8ca51b7de16-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:30:34.093111 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.093082 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" Oct 31 17:30:34.093534 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.093115 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7cw78d" event={"ID":"f5672ab6-0efe-46ba-ac22-b8ca51b7de16","Type":"ContainerDied","Data":"52679a1bc9fbccf31e7c6cd09335012ce43cff58cc3c7be3431126efe4157a62"} Oct 31 17:30:34.093534 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.093165 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52679a1bc9fbccf31e7c6cd09335012ce43cff58cc3c7be3431126efe4157a62" Oct 31 17:30:34.218630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.218601 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:34.322870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.322835 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b2a7d1-4ee9-429d-8665-3bce025cf287-kube-api-access-m6k5z\" (UniqueName: \"kubernetes.io/projected/f9b2a7d1-4ee9-429d-8665-3bce025cf287-kube-api-access-m6k5z\") pod \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\" (UID: \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\") " Oct 31 17:30:34.323043 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.322924 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-util\" (UniqueName: \"kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-util\") pod \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\" (UID: \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\") " Oct 31 17:30:34.323043 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.322981 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-bundle\") pod \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\" (UID: \"f9b2a7d1-4ee9-429d-8665-3bce025cf287\") " Oct 31 17:30:34.323533 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.323507 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-bundle" pod "f9b2a7d1-4ee9-429d-8665-3bce025cf287" (UID: "f9b2a7d1-4ee9-429d-8665-3bce025cf287"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:30:34.325086 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.325066 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b2a7d1-4ee9-429d-8665-3bce025cf287-kube-api-access-m6k5z" pod "f9b2a7d1-4ee9-429d-8665-3bce025cf287" (UID: "f9b2a7d1-4ee9-429d-8665-3bce025cf287"). InnerVolumeSpecName "kube-api-access-m6k5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:30:34.326779 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.326757 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-util" pod "f9b2a7d1-4ee9-429d-8665-3bce025cf287" (UID: "f9b2a7d1-4ee9-429d-8665-3bce025cf287"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:30:34.423855 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.423819 7618 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:30:34.423855 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.423845 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6k5z\" (UniqueName: \"kubernetes.io/projected/f9b2a7d1-4ee9-429d-8665-3bce025cf287-kube-api-access-m6k5z\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:30:34.423855 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:34.423855 7618 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9b2a7d1-4ee9-429d-8665-3bce025cf287-util\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:30:35.099170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:35.099132 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" event={"ID":"f9b2a7d1-4ee9-429d-8665-3bce025cf287","Type":"ContainerDied","Data":"d473d43382abcdf1c89e17dc4f9c9ac794bf9234fb90b6fbb78acb6913b89bd4"} Oct 31 17:30:35.099585 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:35.099181 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d473d43382abcdf1c89e17dc4f9c9ac794bf9234fb90b6fbb78acb6913b89bd4" Oct 31 17:30:35.099585 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:35.099142 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmd6rp" Oct 31 17:30:38.572786 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:30:38.572752 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:30:39.344517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344479 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5"] Oct 31 17:30:39.344817 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344805 7618 cpu_manager.go:414] "containerMap: removing container" podUID="e6007407-99e6-4735-8ee5-b20a10ed0205" containerName="pull" Oct 31 17:30:39.344864 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344819 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6007407-99e6-4735-8ee5-b20a10ed0205" containerName="pull" Oct 31 17:30:39.344864 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344836 7618 cpu_manager.go:414] "containerMap: removing container" podUID="e6007407-99e6-4735-8ee5-b20a10ed0205" containerName="util" Oct 31 17:30:39.344864 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344844 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6007407-99e6-4735-8ee5-b20a10ed0205" containerName="util" Oct 31 17:30:39.344864 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344853 7618 cpu_manager.go:414] "containerMap: removing container" podUID="e6007407-99e6-4735-8ee5-b20a10ed0205" containerName="extract" Oct 31 17:30:39.344864 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344858 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6007407-99e6-4735-8ee5-b20a10ed0205" containerName="extract" Oct 31 17:30:39.345007 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344869 7618 cpu_manager.go:414] "containerMap: removing container" podUID="f9b2a7d1-4ee9-429d-8665-3bce025cf287" containerName="pull" Oct 31 17:30:39.345007 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344933 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b2a7d1-4ee9-429d-8665-3bce025cf287" containerName="pull" Oct 31 17:30:39.345007 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344956 7618 cpu_manager.go:414] "containerMap: removing container" podUID="f5672ab6-0efe-46ba-ac22-b8ca51b7de16" containerName="pull" Oct 31 17:30:39.345007 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344971 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5672ab6-0efe-46ba-ac22-b8ca51b7de16" containerName="pull" Oct 31 17:30:39.345007 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344985 7618 cpu_manager.go:414] "containerMap: removing container" podUID="f9b2a7d1-4ee9-429d-8665-3bce025cf287" containerName="extract" Oct 31 17:30:39.345007 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.344992 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b2a7d1-4ee9-429d-8665-3bce025cf287" containerName="extract" Oct 31 17:30:39.345007 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.345004 7618 cpu_manager.go:414] "containerMap: removing container" podUID="f9b2a7d1-4ee9-429d-8665-3bce025cf287" containerName="util" Oct 31 17:30:39.345007 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.345009 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b2a7d1-4ee9-429d-8665-3bce025cf287" containerName="util" Oct 31 17:30:39.345236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.345019 7618 cpu_manager.go:414] "containerMap: removing container" podUID="f5672ab6-0efe-46ba-ac22-b8ca51b7de16" containerName="extract" Oct 31 17:30:39.345236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.345026 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5672ab6-0efe-46ba-ac22-b8ca51b7de16" containerName="extract" Oct 31 17:30:39.345236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.345038 7618 cpu_manager.go:414] "containerMap: removing container" podUID="f5672ab6-0efe-46ba-ac22-b8ca51b7de16" containerName="util" Oct 31 17:30:39.345236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.345044 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5672ab6-0efe-46ba-ac22-b8ca51b7de16" containerName="util" Oct 31 17:30:39.345236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.345120 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6007407-99e6-4735-8ee5-b20a10ed0205" containerName="extract" Oct 31 17:30:39.345236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.345179 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b2a7d1-4ee9-429d-8665-3bce025cf287" containerName="extract" Oct 31 17:30:39.345236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.345192 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5672ab6-0efe-46ba-ac22-b8ca51b7de16" containerName="extract" Oct 31 17:30:39.347207 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.347186 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5" Oct 31 17:30:39.369893 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.369846 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 17:30:39.369893 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.369871 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-xl89l\"" Oct 31 17:30:39.369893 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.369855 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 17:30:39.370846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.370826 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5"] Oct 31 17:30:39.469176 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.469140 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3382717d-451b-47ef-bc5c-6929994bda4a-tmp\") pod \"jobset-operator-674cd59dc5-xvmk5\" (UID: \"3382717d-451b-47ef-bc5c-6929994bda4a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5" Oct 31 17:30:39.469355 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.469187 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzv5x\" (UniqueName: \"kubernetes.io/projected/3382717d-451b-47ef-bc5c-6929994bda4a-kube-api-access-wzv5x\") pod \"jobset-operator-674cd59dc5-xvmk5\" (UID: \"3382717d-451b-47ef-bc5c-6929994bda4a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5" Oct 31 17:30:39.570289 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.570215 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3382717d-451b-47ef-bc5c-6929994bda4a-tmp\") pod \"jobset-operator-674cd59dc5-xvmk5\" (UID: \"3382717d-451b-47ef-bc5c-6929994bda4a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5" Oct 31 17:30:39.570448 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.570308 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzv5x\" (UniqueName: \"kubernetes.io/projected/3382717d-451b-47ef-bc5c-6929994bda4a-kube-api-access-wzv5x\") pod \"jobset-operator-674cd59dc5-xvmk5\" (UID: \"3382717d-451b-47ef-bc5c-6929994bda4a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5" Oct 31 17:30:39.570612 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.570592 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3382717d-451b-47ef-bc5c-6929994bda4a-tmp\") pod \"jobset-operator-674cd59dc5-xvmk5\" (UID: \"3382717d-451b-47ef-bc5c-6929994bda4a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5" Oct 31 17:30:39.587171 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.587139 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzv5x\" (UniqueName: \"kubernetes.io/projected/3382717d-451b-47ef-bc5c-6929994bda4a-kube-api-access-wzv5x\") pod \"jobset-operator-674cd59dc5-xvmk5\" (UID: \"3382717d-451b-47ef-bc5c-6929994bda4a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5" Oct 31 17:30:39.675594 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.675561 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5" Oct 31 17:30:39.965201 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:39.965103 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5"] Oct 31 17:30:39.969783 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:30:39.969756 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3382717d_451b_47ef_bc5c_6929994bda4a.slice/crio-1eda44818b22f8a6c8996c301fe707e571256c767bf52e8889b400129bebc54a WatchSource:0}: Error finding container 1eda44818b22f8a6c8996c301fe707e571256c767bf52e8889b400129bebc54a: Status 404 returned error can't find the container with id 1eda44818b22f8a6c8996c301fe707e571256c767bf52e8889b400129bebc54a Oct 31 17:30:40.119500 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:40.119466 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5" event={"ID":"3382717d-451b-47ef-bc5c-6929994bda4a","Type":"ContainerStarted","Data":"1eda44818b22f8a6c8996c301fe707e571256c767bf52e8889b400129bebc54a"} Oct 31 17:30:43.134503 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:43.134458 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5" event={"ID":"3382717d-451b-47ef-bc5c-6929994bda4a","Type":"ContainerStarted","Data":"9599b8032f63ca8da72d53d0467dea7674a565c9efc58765e4621f4c173cc816"} Oct 31 17:30:43.169702 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:30:43.169641 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xvmk5" podStartSLOduration=1.5880090390000001 podStartE2EDuration="4.169623879s" totalImagesPullingTime="2.58161484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:30:43.168735079 +0000 UTC m=+164.916709543" watchObservedRunningTime="2026-10-31 17:30:43.169623879 +0000 UTC m=+164.917598347" Oct 31 17:31:01.040104 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.040065 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f"] Oct 31 17:31:01.042705 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.042687 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.143096 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.143062 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 17:31:01.143096 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.143064 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 17:31:01.143400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.143149 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wsnj7\"" Oct 31 17:31:01.143400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.143152 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 17:31:01.143400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.143186 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 17:31:01.143400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.143194 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 17:31:01.158771 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.158747 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8f781277-4933-4a95-bf80-b11bbd34adc5-manifests\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.158851 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.158779 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlzg6\" (UniqueName: \"kubernetes.io/projected/8f781277-4933-4a95-bf80-b11bbd34adc5-kube-api-access-wlzg6\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.158851 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.158820 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f781277-4933-4a95-bf80-b11bbd34adc5-tmp\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.158965 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.158939 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8f781277-4933-4a95-bf80-b11bbd34adc5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.159040 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.159020 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f781277-4933-4a95-bf80-b11bbd34adc5-webhook-cert\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.159115 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.159100 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f781277-4933-4a95-bf80-b11bbd34adc5-metrics-cert\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.167355 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.167325 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f"] Oct 31 17:31:01.259741 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.259708 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8f781277-4933-4a95-bf80-b11bbd34adc5-manifests\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.259741 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.259744 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlzg6\" (UniqueName: \"kubernetes.io/projected/8f781277-4933-4a95-bf80-b11bbd34adc5-kube-api-access-wlzg6\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.259976 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.259782 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f781277-4933-4a95-bf80-b11bbd34adc5-tmp\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.259976 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.259804 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8f781277-4933-4a95-bf80-b11bbd34adc5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.259976 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.259833 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f781277-4933-4a95-bf80-b11bbd34adc5-webhook-cert\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.259976 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.259883 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f781277-4933-4a95-bf80-b11bbd34adc5-metrics-cert\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.260198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.260121 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8f781277-4933-4a95-bf80-b11bbd34adc5-manifests\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.262167 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.262137 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f781277-4933-4a95-bf80-b11bbd34adc5-tmp\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.262438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.262418 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f781277-4933-4a95-bf80-b11bbd34adc5-webhook-cert\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.262520 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.262424 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8f781277-4933-4a95-bf80-b11bbd34adc5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.262520 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.262456 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f781277-4933-4a95-bf80-b11bbd34adc5-metrics-cert\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.276952 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.276919 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlzg6\" (UniqueName: \"kubernetes.io/projected/8f781277-4933-4a95-bf80-b11bbd34adc5-kube-api-access-wlzg6\") pod \"opendatahub-operator-controller-manager-75767875db-lj57f\" (UID: \"8f781277-4933-4a95-bf80-b11bbd34adc5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.356473 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.356382 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:01.759322 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:31:01.759284 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f781277_4933_4a95_bf80_b11bbd34adc5.slice/crio-596e8f5939aa0282ba50102e2a33c7d99d03eb85b6ecdd69f6616dec97fd91b7 WatchSource:0}: Error finding container 596e8f5939aa0282ba50102e2a33c7d99d03eb85b6ecdd69f6616dec97fd91b7: Status 404 returned error can't find the container with id 596e8f5939aa0282ba50102e2a33c7d99d03eb85b6ecdd69f6616dec97fd91b7 Oct 31 17:31:01.763939 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:01.763914 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f"] Oct 31 17:31:02.211684 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:02.211636 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" event={"ID":"8f781277-4933-4a95-bf80-b11bbd34adc5","Type":"ContainerStarted","Data":"596e8f5939aa0282ba50102e2a33c7d99d03eb85b6ecdd69f6616dec97fd91b7"} Oct 31 17:31:05.228629 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:05.228587 7618 generic.go:356] "Generic (PLEG): container finished" podID="8f781277-4933-4a95-bf80-b11bbd34adc5" containerID="33ef3c4195b1c637c47cea4f716a8fe4064fb0983a52ead649c5c2a354912409" exitCode=0 Oct 31 17:31:05.228629 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:05.228628 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" event={"ID":"8f781277-4933-4a95-bf80-b11bbd34adc5","Type":"ContainerDied","Data":"33ef3c4195b1c637c47cea4f716a8fe4064fb0983a52ead649c5c2a354912409"} Oct 31 17:31:06.234319 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:06.234276 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" event={"ID":"8f781277-4933-4a95-bf80-b11bbd34adc5","Type":"ContainerStarted","Data":"a2b830b2ba46784d772dc442b9af6d0fbf25ff7e650f44dca701d58d67e2da71"} Oct 31 17:31:06.234713 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:06.234383 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:06.307969 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:06.307913 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" podStartSLOduration=2.777156406 podStartE2EDuration="6.307897534s" totalImagesPullingTime="3.053377052s" totalInitContainerRuntime="477.364076ms" isStatefulPod=false podCreationTimestamp="2026-10-31 17:31:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:31:06.296726563 +0000 UTC m=+188.044701031" watchObservedRunningTime="2026-10-31 17:31:06.307897534 +0000 UTC m=+188.055872001" Oct 31 17:31:08.573479 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:31:08.573449 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:31:12.203296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.203198 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q"] Oct 31 17:31:12.208517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.208497 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" Oct 31 17:31:12.220483 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.220449 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q"] Oct 31 17:31:12.229624 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.229600 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 17:31:12.229624 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.229599 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4m7zn\"" Oct 31 17:31:12.229836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.229655 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 17:31:12.254073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.254034 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6hs6q\" (UID: \"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" Oct 31 17:31:12.254281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.254086 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5znd9\" (UniqueName: \"kubernetes.io/projected/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-kube-api-access-5znd9\") pod \"kuadrant-operator-controller-manager-678d87766f-6hs6q\" (UID: \"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" Oct 31 17:31:12.354547 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.354509 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6hs6q\" (UID: \"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" Oct 31 17:31:12.354547 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.354550 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5znd9\" (UniqueName: \"kubernetes.io/projected/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-kube-api-access-5znd9\") pod \"kuadrant-operator-controller-manager-678d87766f-6hs6q\" (UID: \"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" Oct 31 17:31:12.354925 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.354904 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6hs6q\" (UID: \"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" Oct 31 17:31:12.372114 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.372076 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5znd9\" (UniqueName: \"kubernetes.io/projected/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-kube-api-access-5znd9\") pod \"kuadrant-operator-controller-manager-678d87766f-6hs6q\" (UID: \"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" Oct 31 17:31:12.520458 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.520373 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" Oct 31 17:31:12.841382 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:12.841289 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q"] Oct 31 17:31:12.859187 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:31:12.859149 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a7a2b9_c43d_47f7_a2ee_fb0575d7accb.slice/crio-f2a2d0d5e9f66a0798469fdae46c6396d7732473c3706ac22b3be3ff3c235e69 WatchSource:0}: Error finding container f2a2d0d5e9f66a0798469fdae46c6396d7732473c3706ac22b3be3ff3c235e69: Status 404 returned error can't find the container with id f2a2d0d5e9f66a0798469fdae46c6396d7732473c3706ac22b3be3ff3c235e69 Oct 31 17:31:13.266161 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:13.266121 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" event={"ID":"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb","Type":"ContainerStarted","Data":"f2a2d0d5e9f66a0798469fdae46c6396d7732473c3706ac22b3be3ff3c235e69"} Oct 31 17:31:17.241543 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:17.241513 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-75767875db-lj57f" Oct 31 17:31:18.316406 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.316372 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h"] Oct 31 17:31:18.320375 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.320348 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" Oct 31 17:31:18.345463 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.345410 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 17:31:18.345609 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.345485 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 17:31:18.345609 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.345540 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zvjcj\"" Oct 31 17:31:18.346168 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.346147 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h"] Oct 31 17:31:18.422070 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.422028 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sljm\" (UniqueName: \"kubernetes.io/projected/60303a6b-433f-4b1c-8e2e-19e901fbd933-kube-api-access-8sljm\") pod \"cert-manager-webhook-5dbbbf75cc-qsm6h\" (UID: \"60303a6b-433f-4b1c-8e2e-19e901fbd933\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" Oct 31 17:31:18.422282 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.422208 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60303a6b-433f-4b1c-8e2e-19e901fbd933-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qsm6h\" (UID: \"60303a6b-433f-4b1c-8e2e-19e901fbd933\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" Oct 31 17:31:18.523450 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.523416 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sljm\" (UniqueName: \"kubernetes.io/projected/60303a6b-433f-4b1c-8e2e-19e901fbd933-kube-api-access-8sljm\") pod \"cert-manager-webhook-5dbbbf75cc-qsm6h\" (UID: \"60303a6b-433f-4b1c-8e2e-19e901fbd933\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" Oct 31 17:31:18.523620 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.523504 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60303a6b-433f-4b1c-8e2e-19e901fbd933-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qsm6h\" (UID: \"60303a6b-433f-4b1c-8e2e-19e901fbd933\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" Oct 31 17:31:18.544650 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.544619 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sljm\" (UniqueName: \"kubernetes.io/projected/60303a6b-433f-4b1c-8e2e-19e901fbd933-kube-api-access-8sljm\") pod \"cert-manager-webhook-5dbbbf75cc-qsm6h\" (UID: \"60303a6b-433f-4b1c-8e2e-19e901fbd933\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" Oct 31 17:31:18.545116 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.544910 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60303a6b-433f-4b1c-8e2e-19e901fbd933-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qsm6h\" (UID: \"60303a6b-433f-4b1c-8e2e-19e901fbd933\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" Oct 31 17:31:18.634572 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.634541 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" Oct 31 17:31:18.947368 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:18.947327 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h"] Oct 31 17:31:18.951348 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:31:18.951313 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60303a6b_433f_4b1c_8e2e_19e901fbd933.slice/crio-ce5f5b96e1a400f657050ef47a5fb974ca1cfb38ac76461012e69b586b307706 WatchSource:0}: Error finding container ce5f5b96e1a400f657050ef47a5fb974ca1cfb38ac76461012e69b586b307706: Status 404 returned error can't find the container with id ce5f5b96e1a400f657050ef47a5fb974ca1cfb38ac76461012e69b586b307706 Oct 31 17:31:19.297948 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:19.297860 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" event={"ID":"60303a6b-433f-4b1c-8e2e-19e901fbd933","Type":"ContainerStarted","Data":"ce5f5b96e1a400f657050ef47a5fb974ca1cfb38ac76461012e69b586b307706"} Oct 31 17:31:19.299420 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:19.299396 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" event={"ID":"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb","Type":"ContainerStarted","Data":"a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9"} Oct 31 17:31:19.299501 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:19.299480 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" Oct 31 17:31:19.337297 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:19.337205 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" podStartSLOduration=1.652941354 podStartE2EDuration="7.337185093s" totalImagesPullingTime="5.684243739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:31:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:31:19.336459436 +0000 UTC m=+201.084433903" watchObservedRunningTime="2026-10-31 17:31:19.337185093 +0000 UTC m=+201.085159558" Oct 31 17:31:20.178074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.178029 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz"] Oct 31 17:31:20.182128 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.182090 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" Oct 31 17:31:20.207662 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.207603 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz"] Oct 31 17:31:20.215929 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.215897 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 17:31:20.239077 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.239037 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl95t\" (UniqueName: \"kubernetes.io/projected/83290f73-1c39-48d4-a226-490f6dc869fc-kube-api-access-jl95t\") pod \"kuadrant-console-plugin-66c4fb575d-548jz\" (UID: \"83290f73-1c39-48d4-a226-490f6dc869fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" Oct 31 17:31:20.239300 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.239109 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/83290f73-1c39-48d4-a226-490f6dc869fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-548jz\" (UID: \"83290f73-1c39-48d4-a226-490f6dc869fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" Oct 31 17:31:20.239300 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.239188 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83290f73-1c39-48d4-a226-490f6dc869fc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-548jz\" (UID: \"83290f73-1c39-48d4-a226-490f6dc869fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" Oct 31 17:31:20.244271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.244231 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7brdk\"" Oct 31 17:31:20.244431 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.244384 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 17:31:20.340449 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.340410 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83290f73-1c39-48d4-a226-490f6dc869fc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-548jz\" (UID: \"83290f73-1c39-48d4-a226-490f6dc869fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" Oct 31 17:31:20.340986 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.340511 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl95t\" (UniqueName: \"kubernetes.io/projected/83290f73-1c39-48d4-a226-490f6dc869fc-kube-api-access-jl95t\") pod \"kuadrant-console-plugin-66c4fb575d-548jz\" (UID: \"83290f73-1c39-48d4-a226-490f6dc869fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" Oct 31 17:31:20.340986 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.340563 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/83290f73-1c39-48d4-a226-490f6dc869fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-548jz\" (UID: \"83290f73-1c39-48d4-a226-490f6dc869fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" Oct 31 17:31:20.341232 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.341207 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83290f73-1c39-48d4-a226-490f6dc869fc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-548jz\" (UID: \"83290f73-1c39-48d4-a226-490f6dc869fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" Oct 31 17:31:20.343986 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.343963 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/83290f73-1c39-48d4-a226-490f6dc869fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-548jz\" (UID: \"83290f73-1c39-48d4-a226-490f6dc869fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" Oct 31 17:31:20.369352 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.369269 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl95t\" (UniqueName: \"kubernetes.io/projected/83290f73-1c39-48d4-a226-490f6dc869fc-kube-api-access-jl95t\") pod \"kuadrant-console-plugin-66c4fb575d-548jz\" (UID: \"83290f73-1c39-48d4-a226-490f6dc869fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" Oct 31 17:31:20.497649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.497561 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" Oct 31 17:31:20.809628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:20.808900 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz"] Oct 31 17:31:20.817921 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:31:20.817889 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83290f73_1c39_48d4_a226_490f6dc869fc.slice/crio-87694fcbe63570d3fb9f731c5b217538d6d9a12c014d5e7ef4ce2ea1d72eca06 WatchSource:0}: Error finding container 87694fcbe63570d3fb9f731c5b217538d6d9a12c014d5e7ef4ce2ea1d72eca06: Status 404 returned error can't find the container with id 87694fcbe63570d3fb9f731c5b217538d6d9a12c014d5e7ef4ce2ea1d72eca06 Oct 31 17:31:21.311316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:21.311272 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" event={"ID":"83290f73-1c39-48d4-a226-490f6dc869fc","Type":"ContainerStarted","Data":"87694fcbe63570d3fb9f731c5b217538d6d9a12c014d5e7ef4ce2ea1d72eca06"} Oct 31 17:31:23.330804 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:23.330771 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" event={"ID":"60303a6b-433f-4b1c-8e2e-19e901fbd933","Type":"ContainerStarted","Data":"c3c0f6706dfb07276b7513e93eb4bb5ab1712dcaefeaf6ae6d47b3a165497cdb"} Oct 31 17:31:23.331270 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:23.330886 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" Oct 31 17:31:23.365542 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:23.365484 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" podStartSLOduration=1.752164415 podStartE2EDuration="5.365468713s" totalImagesPullingTime="3.613304298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:31:23.365427481 +0000 UTC m=+205.113401949" watchObservedRunningTime="2026-10-31 17:31:23.365468713 +0000 UTC m=+205.113443179" Oct 31 17:31:27.350074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:27.350037 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" event={"ID":"83290f73-1c39-48d4-a226-490f6dc869fc","Type":"ContainerStarted","Data":"43e65640476439892c8ca8a75d9fc61a824388ea1cfa9381ec2b60ea00bc50fd"} Oct 31 17:31:27.383975 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:27.383922 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-548jz" podStartSLOduration=1.856886373 podStartE2EDuration="7.383908234s" totalImagesPullingTime="5.527021861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:31:27.383230116 +0000 UTC m=+209.131204582" watchObservedRunningTime="2026-10-31 17:31:27.383908234 +0000 UTC m=+209.131882701" Oct 31 17:31:28.066236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.066190 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l"] Oct 31 17:31:28.070129 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.070103 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" Oct 31 17:31:28.092436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.092409 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 17:31:28.092436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.092418 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 17:31:28.099960 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.099932 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l"] Oct 31 17:31:28.121171 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.121141 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff97ed13-67d4-4416-8246-5b99310096b0-tmp\") pod \"kube-auth-proxy-77485b4dcc-56r9l\" (UID: \"ff97ed13-67d4-4416-8246-5b99310096b0\") " pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" Oct 31 17:31:28.121352 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.121190 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff97ed13-67d4-4416-8246-5b99310096b0-tls-certs\") pod \"kube-auth-proxy-77485b4dcc-56r9l\" (UID: \"ff97ed13-67d4-4416-8246-5b99310096b0\") " pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" Oct 31 17:31:28.121352 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.121317 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scrg9\" (UniqueName: \"kubernetes.io/projected/ff97ed13-67d4-4416-8246-5b99310096b0-kube-api-access-scrg9\") pod \"kube-auth-proxy-77485b4dcc-56r9l\" (UID: \"ff97ed13-67d4-4416-8246-5b99310096b0\") " pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" Oct 31 17:31:28.164456 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.164427 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 17:31:28.164646 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.164556 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 17:31:28.164646 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.164626 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5lqk4\"" Oct 31 17:31:28.222815 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.222778 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scrg9\" (UniqueName: \"kubernetes.io/projected/ff97ed13-67d4-4416-8246-5b99310096b0-kube-api-access-scrg9\") pod \"kube-auth-proxy-77485b4dcc-56r9l\" (UID: \"ff97ed13-67d4-4416-8246-5b99310096b0\") " pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" Oct 31 17:31:28.223005 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.222896 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff97ed13-67d4-4416-8246-5b99310096b0-tmp\") pod \"kube-auth-proxy-77485b4dcc-56r9l\" (UID: \"ff97ed13-67d4-4416-8246-5b99310096b0\") " pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" Oct 31 17:31:28.223005 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.222965 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff97ed13-67d4-4416-8246-5b99310096b0-tls-certs\") pod \"kube-auth-proxy-77485b4dcc-56r9l\" (UID: \"ff97ed13-67d4-4416-8246-5b99310096b0\") " pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" Oct 31 17:31:28.225607 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.225580 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff97ed13-67d4-4416-8246-5b99310096b0-tmp\") pod \"kube-auth-proxy-77485b4dcc-56r9l\" (UID: \"ff97ed13-67d4-4416-8246-5b99310096b0\") " pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" Oct 31 17:31:28.225758 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.225645 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff97ed13-67d4-4416-8246-5b99310096b0-tls-certs\") pod \"kube-auth-proxy-77485b4dcc-56r9l\" (UID: \"ff97ed13-67d4-4416-8246-5b99310096b0\") " pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" Oct 31 17:31:28.240011 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.239984 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scrg9\" (UniqueName: \"kubernetes.io/projected/ff97ed13-67d4-4416-8246-5b99310096b0-kube-api-access-scrg9\") pod \"kube-auth-proxy-77485b4dcc-56r9l\" (UID: \"ff97ed13-67d4-4416-8246-5b99310096b0\") " pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" Oct 31 17:31:28.383426 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.383345 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" Oct 31 17:31:28.702120 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:28.702071 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l"] Oct 31 17:31:28.706405 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:31:28.706373 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff97ed13_67d4_4416_8246_5b99310096b0.slice/crio-aa88c921491b3064ffd0f9e856952544f63c887b2a32a72148adb97cd19a48a6 WatchSource:0}: Error finding container aa88c921491b3064ffd0f9e856952544f63c887b2a32a72148adb97cd19a48a6: Status 404 returned error can't find the container with id aa88c921491b3064ffd0f9e856952544f63c887b2a32a72148adb97cd19a48a6 Oct 31 17:31:29.040009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.039927 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-npzmv"] Oct 31 17:31:29.044718 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.044696 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-npzmv" Oct 31 17:31:29.057761 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.057740 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jvk7n\"" Oct 31 17:31:29.058126 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.058105 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-npzmv"] Oct 31 17:31:29.132332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.132297 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsspp\" (UniqueName: \"kubernetes.io/projected/c8916f81-82eb-4f2b-aa5c-991d268b4600-kube-api-access-jsspp\") pod \"authorino-57f7f8659f-npzmv\" (UID: \"c8916f81-82eb-4f2b-aa5c-991d268b4600\") " pod="kuadrant-system/authorino-57f7f8659f-npzmv" Oct 31 17:31:29.236337 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.234642 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsspp\" (UniqueName: \"kubernetes.io/projected/c8916f81-82eb-4f2b-aa5c-991d268b4600-kube-api-access-jsspp\") pod \"authorino-57f7f8659f-npzmv\" (UID: \"c8916f81-82eb-4f2b-aa5c-991d268b4600\") " pod="kuadrant-system/authorino-57f7f8659f-npzmv" Oct 31 17:31:29.260011 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.259969 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsspp\" (UniqueName: \"kubernetes.io/projected/c8916f81-82eb-4f2b-aa5c-991d268b4600-kube-api-access-jsspp\") pod \"authorino-57f7f8659f-npzmv\" (UID: \"c8916f81-82eb-4f2b-aa5c-991d268b4600\") " pod="kuadrant-system/authorino-57f7f8659f-npzmv" Oct 31 17:31:29.321343 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.319463 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:31:29.323677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.323652 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" Oct 31 17:31:29.339732 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.339707 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" Oct 31 17:31:29.342235 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.342191 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 17:31:29.344149 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.344124 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:31:29.356803 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.356776 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-npzmv" Oct 31 17:31:29.362778 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.362742 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" event={"ID":"ff97ed13-67d4-4416-8246-5b99310096b0","Type":"ContainerStarted","Data":"aa88c921491b3064ffd0f9e856952544f63c887b2a32a72148adb97cd19a48a6"} Oct 31 17:31:29.401697 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.401651 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:31:29.437287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.437071 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/165b216f-bc57-419e-ad86-fe80b6d25168-config-file\") pod \"limitador-limitador-7c9b87d574-8584d\" (UID: \"165b216f-bc57-419e-ad86-fe80b6d25168\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" Oct 31 17:31:29.437287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.437159 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxldb\" (UniqueName: \"kubernetes.io/projected/165b216f-bc57-419e-ad86-fe80b6d25168-kube-api-access-hxldb\") pod \"limitador-limitador-7c9b87d574-8584d\" (UID: \"165b216f-bc57-419e-ad86-fe80b6d25168\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" Oct 31 17:31:29.538624 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.538573 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxldb\" (UniqueName: \"kubernetes.io/projected/165b216f-bc57-419e-ad86-fe80b6d25168-kube-api-access-hxldb\") pod \"limitador-limitador-7c9b87d574-8584d\" (UID: \"165b216f-bc57-419e-ad86-fe80b6d25168\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" Oct 31 17:31:29.538793 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.538727 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/165b216f-bc57-419e-ad86-fe80b6d25168-config-file\") pod \"limitador-limitador-7c9b87d574-8584d\" (UID: \"165b216f-bc57-419e-ad86-fe80b6d25168\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" Oct 31 17:31:29.539516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.539457 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/165b216f-bc57-419e-ad86-fe80b6d25168-config-file\") pod \"limitador-limitador-7c9b87d574-8584d\" (UID: \"165b216f-bc57-419e-ad86-fe80b6d25168\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" Oct 31 17:31:29.557150 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.557109 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxldb\" (UniqueName: \"kubernetes.io/projected/165b216f-bc57-419e-ad86-fe80b6d25168-kube-api-access-hxldb\") pod \"limitador-limitador-7c9b87d574-8584d\" (UID: \"165b216f-bc57-419e-ad86-fe80b6d25168\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" Oct 31 17:31:29.642549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.642515 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" Oct 31 17:31:29.675779 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.675742 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-npzmv"] Oct 31 17:31:29.960414 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:29.960135 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:31:29.965551 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:31:29.965520 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165b216f_bc57_419e_ad86_fe80b6d25168.slice/crio-fbe23c4ca34fb9b312d224386a00ffa39e783750da67b38bbfe5ad9a841cc74c WatchSource:0}: Error finding container fbe23c4ca34fb9b312d224386a00ffa39e783750da67b38bbfe5ad9a841cc74c: Status 404 returned error can't find the container with id fbe23c4ca34fb9b312d224386a00ffa39e783750da67b38bbfe5ad9a841cc74c Oct 31 17:31:30.307282 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:30.307171 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" Oct 31 17:31:30.375999 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:30.375915 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-npzmv" event={"ID":"c8916f81-82eb-4f2b-aa5c-991d268b4600","Type":"ContainerStarted","Data":"32d990a6959d406c0062766fa319f887b349f0d79e0b7b39cec7df0a16d8617e"} Oct 31 17:31:30.377694 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:30.377658 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" event={"ID":"165b216f-bc57-419e-ad86-fe80b6d25168","Type":"ContainerStarted","Data":"fbe23c4ca34fb9b312d224386a00ffa39e783750da67b38bbfe5ad9a841cc74c"} Oct 31 17:31:31.384528 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:31.384443 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" event={"ID":"ff97ed13-67d4-4416-8246-5b99310096b0","Type":"ContainerStarted","Data":"430c8782a4daa27af381ef9ba28cf7c0a4c347a53bfc0f991ae014ace384f57b"} Oct 31 17:31:31.385953 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:31.385927 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" event={"ID":"165b216f-bc57-419e-ad86-fe80b6d25168","Type":"ContainerStarted","Data":"1bc1f5b3a9634b4bde881cc660d503fef3a777fa364523441ee3a039aff83251"} Oct 31 17:31:31.386075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:31.385980 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" Oct 31 17:31:31.433278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:31.433192 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77485b4dcc-56r9l" podStartSLOduration=1.024381915 podStartE2EDuration="3.433178862s" totalImagesPullingTime="2.408796947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:31:31.432380999 +0000 UTC m=+213.180355466" watchObservedRunningTime="2026-10-31 17:31:31.433178862 +0000 UTC m=+213.181153328" Oct 31 17:31:33.396882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:33.396846 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-npzmv" event={"ID":"c8916f81-82eb-4f2b-aa5c-991d268b4600","Type":"ContainerStarted","Data":"968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1"} Oct 31 17:31:33.428691 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:33.428613 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-npzmv" podStartSLOduration=1.223133045 podStartE2EDuration="4.428596614s" totalImagesPullingTime="3.205463569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:31:33.428308711 +0000 UTC m=+215.176283176" watchObservedRunningTime="2026-10-31 17:31:33.428596614 +0000 UTC m=+215.176571083" Oct 31 17:31:33.428891 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:33.428709 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" podStartSLOduration=3.107604654 podStartE2EDuration="4.4287018s" totalImagesPullingTime="1.321097146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:31:31.476615759 +0000 UTC m=+213.224590227" watchObservedRunningTime="2026-10-31 17:31:33.4287018 +0000 UTC m=+215.176676268" Oct 31 17:31:33.528412 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:33.528357 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-npzmv"] Oct 31 17:31:35.412776 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:35.412732 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-npzmv" podUID="c8916f81-82eb-4f2b-aa5c-991d268b4600" containerName="authorino" containerID="cri-o://968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1" gracePeriod=30 Oct 31 17:31:35.659230 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:35.659205 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-npzmv" Oct 31 17:31:35.804236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:35.804147 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8916f81-82eb-4f2b-aa5c-991d268b4600-kube-api-access-jsspp\" (UniqueName: \"kubernetes.io/projected/c8916f81-82eb-4f2b-aa5c-991d268b4600-kube-api-access-jsspp\") pod \"c8916f81-82eb-4f2b-aa5c-991d268b4600\" (UID: \"c8916f81-82eb-4f2b-aa5c-991d268b4600\") " Oct 31 17:31:35.806352 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:35.806330 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8916f81-82eb-4f2b-aa5c-991d268b4600-kube-api-access-jsspp" pod "c8916f81-82eb-4f2b-aa5c-991d268b4600" (UID: "c8916f81-82eb-4f2b-aa5c-991d268b4600"). InnerVolumeSpecName "kube-api-access-jsspp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:31:35.905029 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:35.904991 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsspp\" (UniqueName: \"kubernetes.io/projected/c8916f81-82eb-4f2b-aa5c-991d268b4600-kube-api-access-jsspp\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:31:36.418592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:36.418557 7618 generic.go:356] "Generic (PLEG): container finished" podID="c8916f81-82eb-4f2b-aa5c-991d268b4600" containerID="968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1" exitCode=0 Oct 31 17:31:36.419009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:36.418598 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-npzmv" Oct 31 17:31:36.419009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:36.418603 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-npzmv" event={"ID":"c8916f81-82eb-4f2b-aa5c-991d268b4600","Type":"ContainerDied","Data":"968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1"} Oct 31 17:31:36.419009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:36.418696 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-npzmv" event={"ID":"c8916f81-82eb-4f2b-aa5c-991d268b4600","Type":"ContainerDied","Data":"32d990a6959d406c0062766fa319f887b349f0d79e0b7b39cec7df0a16d8617e"} Oct 31 17:31:36.419009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:36.418712 7618 scope.go:122] "RemoveContainer" containerID="968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1" Oct 31 17:31:36.427004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:36.426988 7618 scope.go:122] "RemoveContainer" containerID="968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1" Oct 31 17:31:36.427285 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:31:36.427264 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1\": container with ID starting with 968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1 not found: ID does not exist" containerID="968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1" Oct 31 17:31:36.427399 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:36.427297 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1"} err="failed to get container status \"968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1\": rpc error: code = NotFound desc = could not find container \"968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1\": container with ID starting with 968957a862b5b3e52abf7a58a31dd66d28481d3353888fcc48795caf6826e3c1 not found: ID does not exist" Oct 31 17:31:36.463372 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:36.463343 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-npzmv"] Oct 31 17:31:36.466235 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:36.466212 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-npzmv"] Oct 31 17:31:38.371917 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:38.371881 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8916f81-82eb-4f2b-aa5c-991d268b4600" path="/var/lib/kubelet/pods/c8916f81-82eb-4f2b-aa5c-991d268b4600/volumes" Oct 31 17:31:38.573327 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:31:38.573284 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:31:42.391778 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:31:42.391749 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-8584d" Oct 31 17:32:04.171741 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.171705 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-jspb9"] Oct 31 17:32:04.172192 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.172144 7618 cpu_manager.go:414] "containerMap: removing container" podUID="c8916f81-82eb-4f2b-aa5c-991d268b4600" containerName="authorino" Oct 31 17:32:04.172192 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.172157 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8916f81-82eb-4f2b-aa5c-991d268b4600" containerName="authorino" Oct 31 17:32:04.172334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.172271 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8916f81-82eb-4f2b-aa5c-991d268b4600" containerName="authorino" Oct 31 17:32:04.179330 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.179308 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:04.193308 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.193270 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 17:32:04.193466 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.193384 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 17:32:04.193537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.193500 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 17:32:04.193537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.193529 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8bpqv\"" Oct 31 17:32:04.193643 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.193585 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 17:32:04.193643 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.193601 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 17:32:04.195311 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.195224 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-jspb9"] Oct 31 17:32:04.270266 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.270214 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-metrics-certs\") pod \"maas-controller-579cf7c6d-jspb9\" (UID: \"59c3b978-eb97-4bf8-b644-f698192d5848\") " pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:04.270433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.270372 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pm64\" (UniqueName: \"kubernetes.io/projected/59c3b978-eb97-4bf8-b644-f698192d5848-kube-api-access-7pm64\") pod \"maas-controller-579cf7c6d-jspb9\" (UID: \"59c3b978-eb97-4bf8-b644-f698192d5848\") " pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:04.270433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.270429 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-webhook-cert\") pod \"maas-controller-579cf7c6d-jspb9\" (UID: \"59c3b978-eb97-4bf8-b644-f698192d5848\") " pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:04.371401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.371361 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-metrics-certs\") pod \"maas-controller-579cf7c6d-jspb9\" (UID: \"59c3b978-eb97-4bf8-b644-f698192d5848\") " pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:04.371628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.371487 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pm64\" (UniqueName: \"kubernetes.io/projected/59c3b978-eb97-4bf8-b644-f698192d5848-kube-api-access-7pm64\") pod \"maas-controller-579cf7c6d-jspb9\" (UID: \"59c3b978-eb97-4bf8-b644-f698192d5848\") " pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:04.371628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.371543 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-webhook-cert\") pod \"maas-controller-579cf7c6d-jspb9\" (UID: \"59c3b978-eb97-4bf8-b644-f698192d5848\") " pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:04.374224 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.374200 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-metrics-certs\") pod \"maas-controller-579cf7c6d-jspb9\" (UID: \"59c3b978-eb97-4bf8-b644-f698192d5848\") " pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:04.374354 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.374330 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-webhook-cert\") pod \"maas-controller-579cf7c6d-jspb9\" (UID: \"59c3b978-eb97-4bf8-b644-f698192d5848\") " pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:04.388430 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.388402 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pm64\" (UniqueName: \"kubernetes.io/projected/59c3b978-eb97-4bf8-b644-f698192d5848-kube-api-access-7pm64\") pod \"maas-controller-579cf7c6d-jspb9\" (UID: \"59c3b978-eb97-4bf8-b644-f698192d5848\") " pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:04.491710 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.491621 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:04.492104 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.492073 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58b696d4f-lt9f8"] Oct 31 17:32:04.494513 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.494495 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:04.516453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.516422 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58b696d4f-lt9f8"] Oct 31 17:32:04.573562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.573532 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-metrics-certs\") pod \"maas-controller-58b696d4f-lt9f8\" (UID: \"026f1505-1522-4ee3-b908-5737ab6b6526\") " pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:04.573717 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.573585 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c475l\" (UniqueName: \"kubernetes.io/projected/026f1505-1522-4ee3-b908-5737ab6b6526-kube-api-access-c475l\") pod \"maas-controller-58b696d4f-lt9f8\" (UID: \"026f1505-1522-4ee3-b908-5737ab6b6526\") " pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:04.573717 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.573650 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-webhook-cert\") pod \"maas-controller-58b696d4f-lt9f8\" (UID: \"026f1505-1522-4ee3-b908-5737ab6b6526\") " pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:04.674355 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.674314 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-metrics-certs\") pod \"maas-controller-58b696d4f-lt9f8\" (UID: \"026f1505-1522-4ee3-b908-5737ab6b6526\") " pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:04.674543 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.674360 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c475l\" (UniqueName: \"kubernetes.io/projected/026f1505-1522-4ee3-b908-5737ab6b6526-kube-api-access-c475l\") pod \"maas-controller-58b696d4f-lt9f8\" (UID: \"026f1505-1522-4ee3-b908-5737ab6b6526\") " pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:04.674543 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.674400 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-webhook-cert\") pod \"maas-controller-58b696d4f-lt9f8\" (UID: \"026f1505-1522-4ee3-b908-5737ab6b6526\") " pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:04.676771 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.676753 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-webhook-cert\") pod \"maas-controller-58b696d4f-lt9f8\" (UID: \"026f1505-1522-4ee3-b908-5737ab6b6526\") " pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:04.676826 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.676804 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-metrics-certs\") pod \"maas-controller-58b696d4f-lt9f8\" (UID: \"026f1505-1522-4ee3-b908-5737ab6b6526\") " pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:04.694267 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.694230 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c475l\" (UniqueName: \"kubernetes.io/projected/026f1505-1522-4ee3-b908-5737ab6b6526-kube-api-access-c475l\") pod \"maas-controller-58b696d4f-lt9f8\" (UID: \"026f1505-1522-4ee3-b908-5737ab6b6526\") " pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:04.793440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.793343 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-jspb9"] Oct 31 17:32:04.799858 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:32:04.799829 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c3b978_eb97_4bf8_b644_f698192d5848.slice/crio-121c0f35c54944e3d97f6571299b79f25d3f65e0f7f150aec44bf72f7f05898f WatchSource:0}: Error finding container 121c0f35c54944e3d97f6571299b79f25d3f65e0f7f150aec44bf72f7f05898f: Status 404 returned error can't find the container with id 121c0f35c54944e3d97f6571299b79f25d3f65e0f7f150aec44bf72f7f05898f Oct 31 17:32:04.825313 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:04.825282 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:05.118310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:05.118176 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58b696d4f-lt9f8"] Oct 31 17:32:05.125281 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:32:05.123072 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026f1505_1522_4ee3_b908_5737ab6b6526.slice/crio-c42309d7fc5c7798470c3927f934fbb67d982bfd0dcf2e2f681a3f600903494e WatchSource:0}: Error finding container c42309d7fc5c7798470c3927f934fbb67d982bfd0dcf2e2f681a3f600903494e: Status 404 returned error can't find the container with id c42309d7fc5c7798470c3927f934fbb67d982bfd0dcf2e2f681a3f600903494e Oct 31 17:32:05.572467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:05.572423 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b696d4f-lt9f8" event={"ID":"026f1505-1522-4ee3-b908-5737ab6b6526","Type":"ContainerStarted","Data":"c42309d7fc5c7798470c3927f934fbb67d982bfd0dcf2e2f681a3f600903494e"} Oct 31 17:32:05.574627 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:05.574593 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-jspb9" event={"ID":"59c3b978-eb97-4bf8-b644-f698192d5848","Type":"ContainerStarted","Data":"121c0f35c54944e3d97f6571299b79f25d3f65e0f7f150aec44bf72f7f05898f"} Oct 31 17:32:08.574236 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:32:08.574202 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:32:08.591662 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:08.591619 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b696d4f-lt9f8" event={"ID":"026f1505-1522-4ee3-b908-5737ab6b6526","Type":"ContainerStarted","Data":"8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73"} Oct 31 17:32:08.591818 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:08.591687 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:08.593032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:08.593007 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-jspb9" event={"ID":"59c3b978-eb97-4bf8-b644-f698192d5848","Type":"ContainerStarted","Data":"4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d"} Oct 31 17:32:08.593146 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:08.593130 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:08.638142 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:08.638082 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-58b696d4f-lt9f8" podStartSLOduration=1.933039408 podStartE2EDuration="4.638068122s" totalImagesPullingTime="2.705028714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:32:08.626832496 +0000 UTC m=+250.374806986" watchObservedRunningTime="2026-10-31 17:32:08.638068122 +0000 UTC m=+250.386042586" Oct 31 17:32:08.662286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:08.662222 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-jspb9" podStartSLOduration=1.873886904 podStartE2EDuration="4.662208177s" totalImagesPullingTime="2.788321273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:32:08.660786554 +0000 UTC m=+250.408761034" watchObservedRunningTime="2026-10-31 17:32:08.662208177 +0000 UTC m=+250.410182692" Oct 31 17:32:09.143190 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.143148 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s"] Oct 31 17:32:09.145835 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.145809 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.162532 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.162508 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 17:32:09.170442 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.170406 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 17:32:09.170682 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.170461 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 17:32:09.170682 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.170643 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-pxmql\"" Oct 31 17:32:09.172640 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.172617 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s"] Oct 31 17:32:09.221231 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.221191 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e77051af-e037-4465-8718-3c8322187994-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.221231 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.221234 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l2sh\" (UniqueName: \"kubernetes.io/projected/e77051af-e037-4465-8718-3c8322187994-kube-api-access-7l2sh\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.221471 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.221358 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e77051af-e037-4465-8718-3c8322187994-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.221471 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.221462 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e77051af-e037-4465-8718-3c8322187994-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.322033 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.322000 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e77051af-e037-4465-8718-3c8322187994-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.322241 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.322059 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e77051af-e037-4465-8718-3c8322187994-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.322241 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.322077 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l2sh\" (UniqueName: \"kubernetes.io/projected/e77051af-e037-4465-8718-3c8322187994-kube-api-access-7l2sh\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.322241 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.322108 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e77051af-e037-4465-8718-3c8322187994-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.322241 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:32:09.322224 7618 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 31 17:32:09.322521 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:32:09.322317 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e77051af-e037-4465-8718-3c8322187994-tls-cert podName:e77051af-e037-4465-8718-3c8322187994 nodeName:}" failed. No retries permitted until 2026-10-31 17:32:09.822299263 +0000 UTC m=+251.570273720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/e77051af-e037-4465-8718-3c8322187994-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-5955s" (UID: "e77051af-e037-4465-8718-3c8322187994") : secret "usage-logs-tenancy-proxy-tls" not found Oct 31 17:32:09.322876 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.322853 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e77051af-e037-4465-8718-3c8322187994-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.322988 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.322968 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e77051af-e037-4465-8718-3c8322187994-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.339330 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.339300 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l2sh\" (UniqueName: \"kubernetes.io/projected/e77051af-e037-4465-8718-3c8322187994-kube-api-access-7l2sh\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.825897 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.825854 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e77051af-e037-4465-8718-3c8322187994-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:09.828316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:09.828291 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e77051af-e037-4465-8718-3c8322187994-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5955s\" (UID: \"e77051af-e037-4465-8718-3c8322187994\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:10.058893 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:10.058837 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:10.352276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:10.352213 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s"] Oct 31 17:32:10.357016 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:32:10.356953 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77051af_e037_4465_8718_3c8322187994.slice/crio-43681bd2e65dd827a93a8bc95b39a71520e9d1b24b4a74e895113b488fb44326 WatchSource:0}: Error finding container 43681bd2e65dd827a93a8bc95b39a71520e9d1b24b4a74e895113b488fb44326: Status 404 returned error can't find the container with id 43681bd2e65dd827a93a8bc95b39a71520e9d1b24b4a74e895113b488fb44326 Oct 31 17:32:10.603932 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:10.603895 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" event={"ID":"e77051af-e037-4465-8718-3c8322187994","Type":"ContainerStarted","Data":"43681bd2e65dd827a93a8bc95b39a71520e9d1b24b4a74e895113b488fb44326"} Oct 31 17:32:19.601414 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:19.601381 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:32:19.602219 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:19.601709 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:19.662452 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:19.662418 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-jspb9"] Oct 31 17:32:19.662865 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:19.662834 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-jspb9" podUID="59c3b978-eb97-4bf8-b644-f698192d5848" containerName="manager" containerID="cri-o://4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d" gracePeriod=30 Oct 31 17:32:19.938075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:19.938048 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:20.029502 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.029458 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-webhook-cert\" (UniqueName: \"kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-webhook-cert\") pod \"59c3b978-eb97-4bf8-b644-f698192d5848\" (UID: \"59c3b978-eb97-4bf8-b644-f698192d5848\") " Oct 31 17:32:20.029691 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.029540 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c3b978-eb97-4bf8-b644-f698192d5848-kube-api-access-7pm64\" (UniqueName: \"kubernetes.io/projected/59c3b978-eb97-4bf8-b644-f698192d5848-kube-api-access-7pm64\") pod \"59c3b978-eb97-4bf8-b644-f698192d5848\" (UID: \"59c3b978-eb97-4bf8-b644-f698192d5848\") " Oct 31 17:32:20.029691 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.029577 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-metrics-certs\") pod \"59c3b978-eb97-4bf8-b644-f698192d5848\" (UID: \"59c3b978-eb97-4bf8-b644-f698192d5848\") " Oct 31 17:32:20.031967 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.031929 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-metrics-certs" pod "59c3b978-eb97-4bf8-b644-f698192d5848" (UID: "59c3b978-eb97-4bf8-b644-f698192d5848"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:32:20.032106 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.032015 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c3b978-eb97-4bf8-b644-f698192d5848-kube-api-access-7pm64" pod "59c3b978-eb97-4bf8-b644-f698192d5848" (UID: "59c3b978-eb97-4bf8-b644-f698192d5848"). InnerVolumeSpecName "kube-api-access-7pm64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:32:20.032280 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.032218 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-webhook-cert" pod "59c3b978-eb97-4bf8-b644-f698192d5848" (UID: "59c3b978-eb97-4bf8-b644-f698192d5848"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:32:20.130800 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.130762 7618 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:32:20.130800 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.130796 7618 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59c3b978-eb97-4bf8-b644-f698192d5848-webhook-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:32:20.130800 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.130807 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pm64\" (UniqueName: \"kubernetes.io/projected/59c3b978-eb97-4bf8-b644-f698192d5848-kube-api-access-7pm64\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:32:20.666119 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.666073 7618 generic.go:356] "Generic (PLEG): container finished" podID="59c3b978-eb97-4bf8-b644-f698192d5848" containerID="4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d" exitCode=0 Oct 31 17:32:20.666638 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.666151 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-jspb9" Oct 31 17:32:20.666638 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.666153 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-jspb9" event={"ID":"59c3b978-eb97-4bf8-b644-f698192d5848","Type":"ContainerDied","Data":"4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d"} Oct 31 17:32:20.666638 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.666219 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-jspb9" event={"ID":"59c3b978-eb97-4bf8-b644-f698192d5848","Type":"ContainerDied","Data":"121c0f35c54944e3d97f6571299b79f25d3f65e0f7f150aec44bf72f7f05898f"} Oct 31 17:32:20.666638 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.666275 7618 scope.go:122] "RemoveContainer" containerID="4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d" Oct 31 17:32:20.677314 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.677285 7618 scope.go:122] "RemoveContainer" containerID="4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d" Oct 31 17:32:20.677676 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:32:20.677657 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d\": container with ID starting with 4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d not found: ID does not exist" containerID="4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d" Oct 31 17:32:20.677771 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.677688 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d"} err="failed to get container status \"4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d\": rpc error: code = NotFound desc = could not find container \"4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d\": container with ID starting with 4cb2f48b94fba43535d8e44caf6cc3218f5aa6e7c16ad53c307e1c9f3c12328d not found: ID does not exist" Oct 31 17:32:20.709775 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.709739 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-jspb9"] Oct 31 17:32:20.720336 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:20.720306 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-jspb9"] Oct 31 17:32:21.673387 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:21.673347 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" event={"ID":"e77051af-e037-4465-8718-3c8322187994","Type":"ContainerStarted","Data":"405a2f0f40ebf23bc1185f3b01064b52cdd32787bff3e7adecb4db96f5e40890"} Oct 31 17:32:21.673772 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:21.673447 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:21.706894 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:21.706841 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" podStartSLOduration=2.034160036 podStartE2EDuration="12.706827857s" totalImagesPullingTime="10.672667821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:32:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:32:21.706523525 +0000 UTC m=+263.454497989" watchObservedRunningTime="2026-10-31 17:32:21.706827857 +0000 UTC m=+263.454802323" Oct 31 17:32:22.374698 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:22.374661 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c3b978-eb97-4bf8-b644-f698192d5848" path="/var/lib/kubelet/pods/59c3b978-eb97-4bf8-b644-f698192d5848/volumes" Oct 31 17:32:30.777443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.777408 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lprq5"] Oct 31 17:32:30.777979 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.777958 7618 cpu_manager.go:414] "containerMap: removing container" podUID="59c3b978-eb97-4bf8-b644-f698192d5848" containerName="manager" Oct 31 17:32:30.778061 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.777982 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c3b978-eb97-4bf8-b644-f698192d5848" containerName="manager" Oct 31 17:32:30.778141 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.778127 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c3b978-eb97-4bf8-b644-f698192d5848" containerName="manager" Oct 31 17:32:30.782413 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.782390 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:32:30.799187 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.799161 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lprq5"] Oct 31 17:32:30.807780 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.807759 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2wjfp\"" Oct 31 17:32:30.810560 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.810542 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 17:32:30.821024 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.820989 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj"] Oct 31 17:32:30.824621 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.824599 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" Oct 31 17:32:30.831448 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.831422 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24hkl\" (UniqueName: \"kubernetes.io/projected/28290dd1-c75b-4c13-88ab-2f9a6dea4795-kube-api-access-24hkl\") pod \"kserve-controller-manager-776cc66d47-lprq5\" (UID: \"28290dd1-c75b-4c13-88ab-2f9a6dea4795\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:32:30.831555 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.831480 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ssd\" (UniqueName: \"kubernetes.io/projected/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-kube-api-access-c5ssd\") pod \"llmisvc-controller-manager-869d5dcc5d-lbsmj\" (UID: \"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" Oct 31 17:32:30.831602 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.831567 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28290dd1-c75b-4c13-88ab-2f9a6dea4795-cert\") pod \"kserve-controller-manager-776cc66d47-lprq5\" (UID: \"28290dd1-c75b-4c13-88ab-2f9a6dea4795\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:32:30.831637 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.831620 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lbsmj\" (UID: \"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" Oct 31 17:32:30.842976 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.842946 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj"] Oct 31 17:32:30.857078 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.857043 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-b4z8w\"" Oct 31 17:32:30.857230 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.857210 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 17:32:30.868618 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.868588 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6bncn"] Oct 31 17:32:30.872472 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.872452 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" Oct 31 17:32:30.890458 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.890433 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6bncn"] Oct 31 17:32:30.900347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.900323 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-6wz7x\"" Oct 31 17:32:30.900347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.900323 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 17:32:30.908623 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.908594 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-r6mcs"] Oct 31 17:32:30.912390 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.912368 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:30.931318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.931289 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-r6mcs"] Oct 31 17:32:30.932297 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.932238 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hkl\" (UniqueName: \"kubernetes.io/projected/28290dd1-c75b-4c13-88ab-2f9a6dea4795-kube-api-access-24hkl\") pod \"kserve-controller-manager-776cc66d47-lprq5\" (UID: \"28290dd1-c75b-4c13-88ab-2f9a6dea4795\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:32:30.932448 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.932351 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ssd\" (UniqueName: \"kubernetes.io/projected/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-kube-api-access-c5ssd\") pod \"llmisvc-controller-manager-869d5dcc5d-lbsmj\" (UID: \"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" Oct 31 17:32:30.932448 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.932404 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9da2c45-46f7-4b1a-8607-069282d70b2d-metrics-certs\") pod \"odh-model-controller-769875db56-r6mcs\" (UID: \"e9da2c45-46f7-4b1a-8607-069282d70b2d\") " pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:30.932448 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.932434 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28290dd1-c75b-4c13-88ab-2f9a6dea4795-cert\") pod \"kserve-controller-manager-776cc66d47-lprq5\" (UID: \"28290dd1-c75b-4c13-88ab-2f9a6dea4795\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:32:30.932608 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.932463 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9da2c45-46f7-4b1a-8607-069282d70b2d-cert\") pod \"odh-model-controller-769875db56-r6mcs\" (UID: \"e9da2c45-46f7-4b1a-8607-069282d70b2d\") " pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:30.932608 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.932487 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d-tls-certs\") pod \"model-serving-api-75bd89fdfc-6bncn\" (UID: \"4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d\") " pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" Oct 31 17:32:30.932608 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.932511 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2fzx\" (UniqueName: \"kubernetes.io/projected/4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d-kube-api-access-h2fzx\") pod \"model-serving-api-75bd89fdfc-6bncn\" (UID: \"4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d\") " pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" Oct 31 17:32:30.932608 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.932570 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lbsmj\" (UID: \"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" Oct 31 17:32:30.932608 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.932594 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffzb\" (UniqueName: \"kubernetes.io/projected/e9da2c45-46f7-4b1a-8607-069282d70b2d-kube-api-access-nffzb\") pod \"odh-model-controller-769875db56-r6mcs\" (UID: \"e9da2c45-46f7-4b1a-8607-069282d70b2d\") " pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:30.935125 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.935101 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lbsmj\" (UID: \"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" Oct 31 17:32:30.935318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.935294 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28290dd1-c75b-4c13-88ab-2f9a6dea4795-cert\") pod \"kserve-controller-manager-776cc66d47-lprq5\" (UID: \"28290dd1-c75b-4c13-88ab-2f9a6dea4795\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:32:30.950055 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.950026 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hkl\" (UniqueName: \"kubernetes.io/projected/28290dd1-c75b-4c13-88ab-2f9a6dea4795-kube-api-access-24hkl\") pod \"kserve-controller-manager-776cc66d47-lprq5\" (UID: \"28290dd1-c75b-4c13-88ab-2f9a6dea4795\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:32:30.953467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:30.953445 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ssd\" (UniqueName: \"kubernetes.io/projected/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-kube-api-access-c5ssd\") pod \"llmisvc-controller-manager-869d5dcc5d-lbsmj\" (UID: \"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" Oct 31 17:32:31.001481 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.001451 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dmqsz\"" Oct 31 17:32:31.001677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.001537 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 17:32:31.001677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.001587 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 17:32:31.033069 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.032979 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffzb\" (UniqueName: \"kubernetes.io/projected/e9da2c45-46f7-4b1a-8607-069282d70b2d-kube-api-access-nffzb\") pod \"odh-model-controller-769875db56-r6mcs\" (UID: \"e9da2c45-46f7-4b1a-8607-069282d70b2d\") " pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:31.033233 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.033074 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9da2c45-46f7-4b1a-8607-069282d70b2d-metrics-certs\") pod \"odh-model-controller-769875db56-r6mcs\" (UID: \"e9da2c45-46f7-4b1a-8607-069282d70b2d\") " pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:31.033233 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.033101 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9da2c45-46f7-4b1a-8607-069282d70b2d-cert\") pod \"odh-model-controller-769875db56-r6mcs\" (UID: \"e9da2c45-46f7-4b1a-8607-069282d70b2d\") " pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:31.033233 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.033118 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d-tls-certs\") pod \"model-serving-api-75bd89fdfc-6bncn\" (UID: \"4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d\") " pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" Oct 31 17:32:31.033233 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.033137 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2fzx\" (UniqueName: \"kubernetes.io/projected/4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d-kube-api-access-h2fzx\") pod \"model-serving-api-75bd89fdfc-6bncn\" (UID: \"4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d\") " pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" Oct 31 17:32:31.035637 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.035614 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9da2c45-46f7-4b1a-8607-069282d70b2d-cert\") pod \"odh-model-controller-769875db56-r6mcs\" (UID: \"e9da2c45-46f7-4b1a-8607-069282d70b2d\") " pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:31.035854 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.035833 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d-tls-certs\") pod \"model-serving-api-75bd89fdfc-6bncn\" (UID: \"4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d\") " pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" Oct 31 17:32:31.035900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.035833 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9da2c45-46f7-4b1a-8607-069282d70b2d-metrics-certs\") pod \"odh-model-controller-769875db56-r6mcs\" (UID: \"e9da2c45-46f7-4b1a-8607-069282d70b2d\") " pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:31.049982 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.049948 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2fzx\" (UniqueName: \"kubernetes.io/projected/4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d-kube-api-access-h2fzx\") pod \"model-serving-api-75bd89fdfc-6bncn\" (UID: \"4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d\") " pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" Oct 31 17:32:31.050579 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.050558 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffzb\" (UniqueName: \"kubernetes.io/projected/e9da2c45-46f7-4b1a-8607-069282d70b2d-kube-api-access-nffzb\") pod \"odh-model-controller-769875db56-r6mcs\" (UID: \"e9da2c45-46f7-4b1a-8607-069282d70b2d\") " pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:31.095440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.095406 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:32:31.136767 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.136729 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" Oct 31 17:32:31.185915 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.185880 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" Oct 31 17:32:31.234537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.234502 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:31.390039 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.389983 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lprq5"] Oct 31 17:32:31.448698 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.448611 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj"] Oct 31 17:32:31.461659 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:32:31.461622 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1fa3f4ff_ff42_438f_8a1e_94db3d52cfd8.slice/crio-e7fa11d4f40dce7eeddeccaedb482f261ac5c4be578ac73ba5ccc21761711176 WatchSource:0}: Error finding container e7fa11d4f40dce7eeddeccaedb482f261ac5c4be578ac73ba5ccc21761711176: Status 404 returned error can't find the container with id e7fa11d4f40dce7eeddeccaedb482f261ac5c4be578ac73ba5ccc21761711176 Oct 31 17:32:31.495283 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.495189 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6bncn"] Oct 31 17:32:31.501977 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:32:31.501946 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de841b8_0cdf_4cab_ac1f_fe2df4cdbd9d.slice/crio-8cbd79ca9dd9686139416d45b07ca5d63f7b929dfe6de7c0d6c2d2dccac01b52 WatchSource:0}: Error finding container 8cbd79ca9dd9686139416d45b07ca5d63f7b929dfe6de7c0d6c2d2dccac01b52: Status 404 returned error can't find the container with id 8cbd79ca9dd9686139416d45b07ca5d63f7b929dfe6de7c0d6c2d2dccac01b52 Oct 31 17:32:31.535877 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.535843 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-r6mcs"] Oct 31 17:32:31.540313 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:32:31.540282 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9da2c45_46f7_4b1a_8607_069282d70b2d.slice/crio-2edf44f3dc6896dbbecda091a095bd0356614d97666090a26f75e350c443e7e4 WatchSource:0}: Error finding container 2edf44f3dc6896dbbecda091a095bd0356614d97666090a26f75e350c443e7e4: Status 404 returned error can't find the container with id 2edf44f3dc6896dbbecda091a095bd0356614d97666090a26f75e350c443e7e4 Oct 31 17:32:31.723298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.723234 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-r6mcs" event={"ID":"e9da2c45-46f7-4b1a-8607-069282d70b2d","Type":"ContainerStarted","Data":"2edf44f3dc6896dbbecda091a095bd0356614d97666090a26f75e350c443e7e4"} Oct 31 17:32:31.724634 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.724600 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" event={"ID":"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8","Type":"ContainerStarted","Data":"e7fa11d4f40dce7eeddeccaedb482f261ac5c4be578ac73ba5ccc21761711176"} Oct 31 17:32:31.725778 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.725756 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" event={"ID":"4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d","Type":"ContainerStarted","Data":"8cbd79ca9dd9686139416d45b07ca5d63f7b929dfe6de7c0d6c2d2dccac01b52"} Oct 31 17:32:31.727035 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:31.727014 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" event={"ID":"28290dd1-c75b-4c13-88ab-2f9a6dea4795","Type":"ContainerStarted","Data":"06b24173e4a938162b4c827d32881f3e3765b99361a861eeb8d0bb7ac68284f9"} Oct 31 17:32:32.686005 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:32.685969 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5955s" Oct 31 17:32:34.755392 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:34.755331 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" event={"ID":"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8","Type":"ContainerStarted","Data":"3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd"} Oct 31 17:32:34.755856 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:34.755425 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" Oct 31 17:32:34.757055 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:34.757000 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" event={"ID":"4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d","Type":"ContainerStarted","Data":"8bfc2d7e86a8bcdd15063d576954788c64425f97d7d42880130e1fe4900548b9"} Oct 31 17:32:34.757194 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:34.757130 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" Oct 31 17:32:34.758858 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:34.758831 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" event={"ID":"28290dd1-c75b-4c13-88ab-2f9a6dea4795","Type":"ContainerStarted","Data":"b18d15a17c6620fd1cd0fe118170b2f655771533e86e265c94291d2d1779371c"} Oct 31 17:32:34.758964 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:34.758879 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:32:34.790084 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:34.790021 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" podStartSLOduration=1.652325196 podStartE2EDuration="4.790005455s" totalImagesPullingTime="3.137680259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:32:34.788788622 +0000 UTC m=+276.536763096" watchObservedRunningTime="2026-10-31 17:32:34.790005455 +0000 UTC m=+276.537979921" Oct 31 17:32:34.822309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:34.822232 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" podStartSLOduration=1.8843896070000001 podStartE2EDuration="4.822214006s" totalImagesPullingTime="2.937824399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:32:34.821372447 +0000 UTC m=+276.569346915" watchObservedRunningTime="2026-10-31 17:32:34.822214006 +0000 UTC m=+276.570188475" Oct 31 17:32:34.855957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:34.855850 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" podStartSLOduration=2.177141544 podStartE2EDuration="4.855834306s" totalImagesPullingTime="2.678692762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:32:34.854430368 +0000 UTC m=+276.602404835" watchObservedRunningTime="2026-10-31 17:32:34.855834306 +0000 UTC m=+276.603808773" Oct 31 17:32:35.764927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:35.764887 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-r6mcs" event={"ID":"e9da2c45-46f7-4b1a-8607-069282d70b2d","Type":"ContainerStarted","Data":"ef58724b17d863e637086ccc5f3b9823612f3c9e31cd3e5d1c96623a8e00af9e"} Oct 31 17:32:35.765379 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:35.765147 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:35.800111 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:35.800058 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-r6mcs" podStartSLOduration=2.237962827 podStartE2EDuration="5.800042378s" totalImagesPullingTime="3.562079551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:32:35.799717052 +0000 UTC m=+277.547691520" watchObservedRunningTime="2026-10-31 17:32:35.800042378 +0000 UTC m=+277.548016863" Oct 31 17:32:38.571320 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:32:38.571285 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:32:41.726967 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:41.726930 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:32:45.768889 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:45.768859 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-6bncn" Oct 31 17:32:46.772526 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:46.772493 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-r6mcs" Oct 31 17:32:50.718536 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:50.718499 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f7b7f665d-k5sf7"] Oct 31 17:32:50.976199 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:50.976116 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f7b7f665d-k5sf7"] Oct 31 17:32:50.976373 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:50.976214 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" Oct 31 17:32:50.999302 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:50.999244 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 17:32:50.999302 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:50.999244 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jvk7n\"" Oct 31 17:32:51.121483 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:51.121441 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2sdm\" (UniqueName: \"kubernetes.io/projected/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-kube-api-access-w2sdm\") pod \"authorino-5f7b7f665d-k5sf7\" (UID: \"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c\") " pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" Oct 31 17:32:51.121652 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:51.121566 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-tls-cert\") pod \"authorino-5f7b7f665d-k5sf7\" (UID: \"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c\") " pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" Oct 31 17:32:51.222633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:51.222596 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-tls-cert\") pod \"authorino-5f7b7f665d-k5sf7\" (UID: \"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c\") " pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" Oct 31 17:32:51.222823 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:51.222670 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2sdm\" (UniqueName: \"kubernetes.io/projected/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-kube-api-access-w2sdm\") pod \"authorino-5f7b7f665d-k5sf7\" (UID: \"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c\") " pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" Oct 31 17:32:51.225033 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:51.225003 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-tls-cert\") pod \"authorino-5f7b7f665d-k5sf7\" (UID: \"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c\") " pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" Oct 31 17:32:51.240891 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:51.240823 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2sdm\" (UniqueName: \"kubernetes.io/projected/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-kube-api-access-w2sdm\") pod \"authorino-5f7b7f665d-k5sf7\" (UID: \"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c\") " pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" Oct 31 17:32:51.302610 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:51.302572 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" Oct 31 17:32:51.612405 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:51.612313 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f7b7f665d-k5sf7"] Oct 31 17:32:51.617060 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:32:51.617031 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd105c1bc_0e89_4e82_9bfd_4be5b67b1f7c.slice/crio-bcfe4085791657090d04222bc8ce85d2406c7264ec9825d77effa88e3732ed27 WatchSource:0}: Error finding container bcfe4085791657090d04222bc8ce85d2406c7264ec9825d77effa88e3732ed27: Status 404 returned error can't find the container with id bcfe4085791657090d04222bc8ce85d2406c7264ec9825d77effa88e3732ed27 Oct 31 17:32:51.845741 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:51.845705 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" event={"ID":"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c","Type":"ContainerStarted","Data":"bcfe4085791657090d04222bc8ce85d2406c7264ec9825d77effa88e3732ed27"} Oct 31 17:32:52.858196 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:52.858103 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" event={"ID":"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c","Type":"ContainerStarted","Data":"f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd"} Oct 31 17:32:52.891614 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:52.891551 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" podStartSLOduration=2.04287591 podStartE2EDuration="2.891536957s" totalImagesPullingTime="848.661047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:32:52.890429769 +0000 UTC m=+294.638404236" watchObservedRunningTime="2026-10-31 17:32:52.891536957 +0000 UTC m=+294.639511713" Oct 31 17:32:58.419769 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:58.419741 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:32:58.420164 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:32:58.419787 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:33:05.767280 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:05.767224 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" Oct 31 17:33:05.770173 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:05.770152 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:33:08.572657 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:33:08.572622 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:33:34.171274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:34.171216 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8"] Oct 31 17:33:34.174417 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:34.174395 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" Oct 31 17:33:34.192969 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:34.192940 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8"] Oct 31 17:33:34.217317 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:34.217284 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0672b07e-809a-443d-9dd8-6df70e1e5ddf-cert\") pod \"llmisvc-controller-manager-6466c554d8-2t6x8\" (UID: \"0672b07e-809a-443d-9dd8-6df70e1e5ddf\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" Oct 31 17:33:34.217508 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:34.217331 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q7s9\" (UniqueName: \"kubernetes.io/projected/0672b07e-809a-443d-9dd8-6df70e1e5ddf-kube-api-access-8q7s9\") pod \"llmisvc-controller-manager-6466c554d8-2t6x8\" (UID: \"0672b07e-809a-443d-9dd8-6df70e1e5ddf\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" Oct 31 17:33:34.318860 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:34.318822 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0672b07e-809a-443d-9dd8-6df70e1e5ddf-cert\") pod \"llmisvc-controller-manager-6466c554d8-2t6x8\" (UID: \"0672b07e-809a-443d-9dd8-6df70e1e5ddf\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" Oct 31 17:33:34.318860 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:34.318863 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q7s9\" (UniqueName: \"kubernetes.io/projected/0672b07e-809a-443d-9dd8-6df70e1e5ddf-kube-api-access-8q7s9\") pod \"llmisvc-controller-manager-6466c554d8-2t6x8\" (UID: \"0672b07e-809a-443d-9dd8-6df70e1e5ddf\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" Oct 31 17:33:34.321309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:34.321287 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0672b07e-809a-443d-9dd8-6df70e1e5ddf-cert\") pod \"llmisvc-controller-manager-6466c554d8-2t6x8\" (UID: \"0672b07e-809a-443d-9dd8-6df70e1e5ddf\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" Oct 31 17:33:34.336340 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:34.336310 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q7s9\" (UniqueName: \"kubernetes.io/projected/0672b07e-809a-443d-9dd8-6df70e1e5ddf-kube-api-access-8q7s9\") pod \"llmisvc-controller-manager-6466c554d8-2t6x8\" (UID: \"0672b07e-809a-443d-9dd8-6df70e1e5ddf\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" Oct 31 17:33:34.486402 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:34.486315 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" Oct 31 17:33:34.776181 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:34.776067 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8"] Oct 31 17:33:34.780186 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:33:34.780158 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0672b07e_809a_443d_9dd8_6df70e1e5ddf.slice/crio-c0208e8973c1219228f6d49fa126bd8b6f4dd0655bed6b55a35ce35c39c8e07e WatchSource:0}: Error finding container c0208e8973c1219228f6d49fa126bd8b6f4dd0655bed6b55a35ce35c39c8e07e: Status 404 returned error can't find the container with id c0208e8973c1219228f6d49fa126bd8b6f4dd0655bed6b55a35ce35c39c8e07e Oct 31 17:33:35.074234 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:35.074198 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" event={"ID":"0672b07e-809a-443d-9dd8-6df70e1e5ddf","Type":"ContainerStarted","Data":"c0208e8973c1219228f6d49fa126bd8b6f4dd0655bed6b55a35ce35c39c8e07e"} Oct 31 17:33:36.080859 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:36.080821 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" event={"ID":"0672b07e-809a-443d-9dd8-6df70e1e5ddf","Type":"ContainerStarted","Data":"5679bdbec08db9a9640e999d8e76ddb9705d5dcab6e55c6ca190e3aa64b604b7"} Oct 31 17:33:36.081286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:36.080888 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" Oct 31 17:33:36.116624 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:33:36.116557 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" podStartSLOduration=1.834009534 podStartE2EDuration="2.116533571s" totalImagesPullingTime="282.524037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:33:36.114194933 +0000 UTC m=+337.862169401" watchObservedRunningTime="2026-10-31 17:33:36.116533571 +0000 UTC m=+337.864508042" Oct 31 17:33:38.571634 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:33:38.571595 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:34:07.088135 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:07.088093 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" Oct 31 17:34:07.149720 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:07.149681 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj"] Oct 31 17:34:07.149996 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:07.149954 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" podUID="1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8" containerName="manager" containerID="cri-o://3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd" gracePeriod=30 Oct 31 17:34:07.407555 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:07.407527 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" Oct 31 17:34:07.409490 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:07.409469 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-cert\" (UniqueName: \"kubernetes.io/secret/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-cert\") pod \"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8\" (UID: \"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8\") " Oct 31 17:34:07.409614 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:07.409541 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-kube-api-access-c5ssd\" (UniqueName: \"kubernetes.io/projected/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-kube-api-access-c5ssd\") pod \"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8\" (UID: \"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8\") " Oct 31 17:34:07.411631 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:07.411606 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-cert" pod "1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8" (UID: "1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:34:07.411979 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:07.411945 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-kube-api-access-c5ssd" pod "1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8" (UID: "1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8"). InnerVolumeSpecName "kube-api-access-c5ssd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:34:07.510871 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:07.510833 7618 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:34:07.510871 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:07.510870 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5ssd\" (UniqueName: \"kubernetes.io/projected/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8-kube-api-access-c5ssd\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:34:08.231673 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:08.231637 7618 generic.go:356] "Generic (PLEG): container finished" podID="1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8" containerID="3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd" exitCode=0 Oct 31 17:34:08.232073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:08.231689 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" Oct 31 17:34:08.232073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:08.231727 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" event={"ID":"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8","Type":"ContainerDied","Data":"3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd"} Oct 31 17:34:08.232073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:08.231788 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj" event={"ID":"1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8","Type":"ContainerDied","Data":"e7fa11d4f40dce7eeddeccaedb482f261ac5c4be578ac73ba5ccc21761711176"} Oct 31 17:34:08.232073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:08.231814 7618 scope.go:122] "RemoveContainer" containerID="3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd" Oct 31 17:34:08.241063 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:08.241044 7618 scope.go:122] "RemoveContainer" containerID="3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd" Oct 31 17:34:08.241392 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:34:08.241372 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd\": container with ID starting with 3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd not found: ID does not exist" containerID="3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd" Oct 31 17:34:08.241451 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:08.241402 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd"} err="failed to get container status \"3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd\": rpc error: code = NotFound desc = could not find container \"3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd\": container with ID starting with 3d1888e98ea5ac2f1322d03969dc570f192ec42455e85d5469e3e1f6e2a64dbd not found: ID does not exist" Oct 31 17:34:08.275949 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:08.275909 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj"] Oct 31 17:34:08.286317 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:08.286282 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lbsmj"] Oct 31 17:34:08.372192 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:08.372157 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8" path="/var/lib/kubelet/pods/1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8/volumes" Oct 31 17:34:08.571971 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:34:08.571886 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:34:26.275015 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:26.274931 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:34:29.839264 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:29.839208 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:34:33.288391 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:33.288354 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:34:37.381442 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:37.381407 7618 generic.go:356] "Generic (PLEG): container finished" podID="026f1505-1522-4ee3-b908-5737ab6b6526" containerID="8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73" exitCode=1 Oct 31 17:34:37.381892 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:37.381481 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b696d4f-lt9f8" event={"ID":"026f1505-1522-4ee3-b908-5737ab6b6526","Type":"ContainerDied","Data":"8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73"} Oct 31 17:34:37.381892 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:37.381850 7618 scope.go:122] "RemoveContainer" containerID="8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73" Oct 31 17:34:38.086537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:38.086446 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:34:38.388004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:38.387930 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b696d4f-lt9f8" event={"ID":"026f1505-1522-4ee3-b908-5737ab6b6526","Type":"ContainerStarted","Data":"dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6"} Oct 31 17:34:38.388385 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:38.388132 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:34:38.571458 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:34:38.571420 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:34:41.879060 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:41.879023 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:34:47.172029 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:47.171982 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:34:49.398865 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:49.398830 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:34:57.215690 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.215644 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9"] Oct 31 17:34:57.216206 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.216188 7618 cpu_manager.go:414] "containerMap: removing container" podUID="1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8" containerName="manager" Oct 31 17:34:57.216270 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.216209 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8" containerName="manager" Oct 31 17:34:57.216322 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.216312 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa3f4ff-ff42-438f-8a1e-94db3d52cfd8" containerName="manager" Oct 31 17:34:57.218753 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.218725 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.232520 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.232493 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 17:34:57.248175 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.248138 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9"] Oct 31 17:34:57.290899 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.290852 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.291107 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.290922 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63bb320-851c-4bd6-880b-906a3c12bbda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.291107 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.290993 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63bb320-851c-4bd6-880b-906a3c12bbda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.291107 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.291027 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.291107 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.291054 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.291312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.291115 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.291312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.291145 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63bb320-851c-4bd6-880b-906a3c12bbda-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.291312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.291189 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb7vq\" (UniqueName: \"kubernetes.io/projected/e63bb320-851c-4bd6-880b-906a3c12bbda-kube-api-access-sb7vq\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.291428 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.291316 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.324429 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.324394 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9kmpm\"" Oct 31 17:34:57.392431 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.392392 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.392633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.392442 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63bb320-851c-4bd6-880b-906a3c12bbda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.392633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.392457 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63bb320-851c-4bd6-880b-906a3c12bbda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.392633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.392481 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.392633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.392495 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.392633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.392511 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.392633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.392535 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63bb320-851c-4bd6-880b-906a3c12bbda-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.392633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.392580 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb7vq\" (UniqueName: \"kubernetes.io/projected/e63bb320-851c-4bd6-880b-906a3c12bbda-kube-api-access-sb7vq\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.392633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.392631 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.393060 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.392908 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.393060 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.392934 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.393060 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.393005 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.393301 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.393279 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.393301 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.393291 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63bb320-851c-4bd6-880b-906a3c12bbda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.395507 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.395478 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63bb320-851c-4bd6-880b-906a3c12bbda-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.395625 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.395608 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63bb320-851c-4bd6-880b-906a3c12bbda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.409212 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.409172 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63bb320-851c-4bd6-880b-906a3c12bbda-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.409212 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.409192 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb7vq\" (UniqueName: \"kubernetes.io/projected/e63bb320-851c-4bd6-880b-906a3c12bbda-kube-api-access-sb7vq\") pod \"maas-default-gateway-openshift-default-7d9897b54c-4vwr9\" (UID: \"e63bb320-851c-4bd6-880b-906a3c12bbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.532131 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.532038 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:34:57.839433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:57.839343 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9"] Oct 31 17:34:57.846436 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:34:57.846392 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63bb320_851c_4bd6_880b_906a3c12bbda.slice/crio-99c20042f457f9d315e82bdd09706e7ff23e73ff1df4bdef9de6f0f1abe83726 WatchSource:0}: Error finding container 99c20042f457f9d315e82bdd09706e7ff23e73ff1df4bdef9de6f0f1abe83726: Status 404 returned error can't find the container with id 99c20042f457f9d315e82bdd09706e7ff23e73ff1df4bdef9de6f0f1abe83726 Oct 31 17:34:58.510960 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:58.510926 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" event={"ID":"e63bb320-851c-4bd6-880b-906a3c12bbda","Type":"ContainerStarted","Data":"99c20042f457f9d315e82bdd09706e7ff23e73ff1df4bdef9de6f0f1abe83726"} Oct 31 17:34:58.647819 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:58.647148 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f7b7f665d-k5sf7"] Oct 31 17:34:58.647819 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:58.647372 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" podUID="d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c" containerName="authorino" containerID="cri-o://f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd" gracePeriod=30 Oct 31 17:34:58.953047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:58.953013 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" Oct 31 17:34:59.010134 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.010055 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-tls-cert\" (UniqueName: \"kubernetes.io/secret/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-tls-cert\") pod \"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c\" (UID: \"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c\") " Oct 31 17:34:59.010671 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.010237 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-kube-api-access-w2sdm\" (UniqueName: \"kubernetes.io/projected/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-kube-api-access-w2sdm\") pod \"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c\" (UID: \"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c\") " Oct 31 17:34:59.012837 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.012810 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-kube-api-access-w2sdm" pod "d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c" (UID: "d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c"). InnerVolumeSpecName "kube-api-access-w2sdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:34:59.025525 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.025487 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-tls-cert" pod "d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c" (UID: "d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:34:59.111722 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.111683 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2sdm\" (UniqueName: \"kubernetes.io/projected/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-kube-api-access-w2sdm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:34:59.111722 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.111719 7618 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c-tls-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:34:59.283301 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.283188 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:34:59.518157 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.518120 7618 generic.go:356] "Generic (PLEG): container finished" podID="d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c" containerID="f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd" exitCode=0 Oct 31 17:34:59.518738 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.518167 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" Oct 31 17:34:59.518738 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.518185 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" event={"ID":"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c","Type":"ContainerDied","Data":"f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd"} Oct 31 17:34:59.518738 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.518223 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7b7f665d-k5sf7" event={"ID":"d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c","Type":"ContainerDied","Data":"bcfe4085791657090d04222bc8ce85d2406c7264ec9825d77effa88e3732ed27"} Oct 31 17:34:59.518738 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.518275 7618 scope.go:122] "RemoveContainer" containerID="f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd" Oct 31 17:34:59.533995 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.533974 7618 scope.go:122] "RemoveContainer" containerID="f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd" Oct 31 17:34:59.534388 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:34:59.534357 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd\": container with ID starting with f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd not found: ID does not exist" containerID="f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd" Oct 31 17:34:59.534488 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.534402 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd"} err="failed to get container status \"f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd\": rpc error: code = NotFound desc = could not find container \"f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd\": container with ID starting with f6a33f3be634527988e11cdcc13ddbbdbbd6a44d2cca2c1a89f79afd37026fcd not found: ID does not exist" Oct 31 17:34:59.572917 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.572882 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f7b7f665d-k5sf7"] Oct 31 17:34:59.577122 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:34:59.577087 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f7b7f665d-k5sf7"] Oct 31 17:35:00.373592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:00.373555 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c" path="/var/lib/kubelet/pods/d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c/volumes" Oct 31 17:35:01.040851 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:01.040816 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:35:01.041153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:01.040890 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:35:01.041153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:01.040918 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:35:01.561000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:01.560965 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" event={"ID":"e63bb320-851c-4bd6-880b-906a3c12bbda","Type":"ContainerStarted","Data":"251e0e57906f36d56c1acc5bfeb78e6797c7cb605a79220a4315b45d61c365bc"} Oct 31 17:35:01.609138 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:01.609081 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" podStartSLOduration=1.417838532 podStartE2EDuration="4.609065097s" totalImagesPullingTime="3.191226565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:35:01.6089514 +0000 UTC m=+423.356925868" watchObservedRunningTime="2026-10-31 17:35:01.609065097 +0000 UTC m=+423.357039564" Oct 31 17:35:02.546279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:02.543041 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:35:02.546279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:02.543174 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4vwr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 17:35:02.546279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:02.543221 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" podUID="e63bb320-851c-4bd6-880b-906a3c12bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 17:35:03.534703 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:03.534660 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4vwr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 17:35:03.535054 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:03.535011 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" podUID="e63bb320-851c-4bd6-880b-906a3c12bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 17:35:04.540307 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:04.537661 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4vwr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 17:35:04.540307 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:04.537725 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" podUID="e63bb320-851c-4bd6-880b-906a3c12bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 17:35:05.537276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:05.536638 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4vwr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 17:35:05.537276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:05.536717 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" podUID="e63bb320-851c-4bd6-880b-906a3c12bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 17:35:06.540279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:06.534668 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4vwr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 17:35:06.540279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:06.534747 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" podUID="e63bb320-851c-4bd6-880b-906a3c12bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 17:35:07.533055 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:07.533012 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4vwr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 17:35:07.533302 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:07.533080 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" podUID="e63bb320-851c-4bd6-880b-906a3c12bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 17:35:08.572474 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:35:08.572437 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:35:09.533439 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:09.533397 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4vwr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:35:09.533765 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:09.533453 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" podUID="e63bb320-851c-4bd6-880b-906a3c12bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:35:10.574504 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:10.574450 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4vwr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:35:10.575075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:10.574522 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" podUID="e63bb320-851c-4bd6-880b-906a3c12bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:35:11.334856 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:11.334822 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:35:11.335288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:11.335230 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:35:11.336073 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:11.336052 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" Oct 31 17:35:12.230749 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.230708 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw"] Oct 31 17:35:12.232299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.232245 7618 cpu_manager.go:414] "containerMap: removing container" podUID="d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c" containerName="authorino" Oct 31 17:35:12.232299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.232296 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c" containerName="authorino" Oct 31 17:35:12.232482 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.232422 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="d105c1bc-0e89-4e82-9bfd-4be5b67b1f7c" containerName="authorino" Oct 31 17:35:12.235568 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.235547 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.259678 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.259641 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw"] Oct 31 17:35:12.355111 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.355055 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.355380 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.355134 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df608734-a9f8-40c6-9430-1b25fa3389e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.355380 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.355273 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.355380 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.355312 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df608734-a9f8-40c6-9430-1b25fa3389e4-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.355541 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.355400 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.355541 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.355451 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.355648 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.355542 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.355648 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.355597 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df608734-a9f8-40c6-9430-1b25fa3389e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.355648 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.355629 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z8x8\" (UniqueName: \"kubernetes.io/projected/df608734-a9f8-40c6-9430-1b25fa3389e4-kube-api-access-2z8x8\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.456722 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.456688 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df608734-a9f8-40c6-9430-1b25fa3389e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.456722 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.456725 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z8x8\" (UniqueName: \"kubernetes.io/projected/df608734-a9f8-40c6-9430-1b25fa3389e4-kube-api-access-2z8x8\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.456973 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.456759 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.456973 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.456821 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df608734-a9f8-40c6-9430-1b25fa3389e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.456973 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.456852 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.456973 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.456876 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df608734-a9f8-40c6-9430-1b25fa3389e4-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.456973 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.456903 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.457234 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.456990 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.457234 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.457042 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.457596 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.457568 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.457596 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.457587 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.457732 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.457671 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.457732 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.457710 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df608734-a9f8-40c6-9430-1b25fa3389e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.457732 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.457723 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.459710 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.459690 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df608734-a9f8-40c6-9430-1b25fa3389e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.459964 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.459944 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df608734-a9f8-40c6-9430-1b25fa3389e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.473490 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.473461 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df608734-a9f8-40c6-9430-1b25fa3389e4-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.473867 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.473843 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z8x8\" (UniqueName: \"kubernetes.io/projected/df608734-a9f8-40c6-9430-1b25fa3389e4-kube-api-access-2z8x8\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kd6mw\" (UID: \"df608734-a9f8-40c6-9430-1b25fa3389e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.549345 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.549228 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:12.858894 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.858799 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw"] Oct 31 17:35:12.862962 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:35:12.862932 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf608734_a9f8_40c6_9430_1b25fa3389e4.slice/crio-b008aab494e7438a1ca903ff8da04356ae4e33069d2fbac08f205a2114080dc0 WatchSource:0}: Error finding container b008aab494e7438a1ca903ff8da04356ae4e33069d2fbac08f205a2114080dc0: Status 404 returned error can't find the container with id b008aab494e7438a1ca903ff8da04356ae4e33069d2fbac08f205a2114080dc0 Oct 31 17:35:12.865964 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.865934 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:35:12.866051 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.865995 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:35:12.866051 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:12.866028 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:35:13.645943 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:13.645906 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" event={"ID":"df608734-a9f8-40c6-9430-1b25fa3389e4","Type":"ContainerStarted","Data":"f5deec153042ffdae6adcd192be0dabeb57284cd3295cc3a15134e65cdc80c8b"} Oct 31 17:35:13.646353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:13.645971 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" event={"ID":"df608734-a9f8-40c6-9430-1b25fa3389e4","Type":"ContainerStarted","Data":"b008aab494e7438a1ca903ff8da04356ae4e33069d2fbac08f205a2114080dc0"} Oct 31 17:35:14.549447 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:14.549414 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:14.550832 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:14.550803 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kd6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 17:35:14.550964 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:14.550850 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" podUID="df608734-a9f8-40c6-9430-1b25fa3389e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 17:35:15.549858 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:15.549816 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kd6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 17:35:15.550291 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:15.549890 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" podUID="df608734-a9f8-40c6-9430-1b25fa3389e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 17:35:16.549798 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:16.549762 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kd6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 17:35:16.550011 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:16.549821 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" podUID="df608734-a9f8-40c6-9430-1b25fa3389e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 17:35:17.549883 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:17.549843 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kd6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 17:35:17.550072 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:17.549905 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" podUID="df608734-a9f8-40c6-9430-1b25fa3389e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 17:35:18.550585 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:18.550548 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kd6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 17:35:18.551093 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:18.550610 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" podUID="df608734-a9f8-40c6-9430-1b25fa3389e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 17:35:19.549847 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:19.549809 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kd6mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 31 17:35:19.550048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:19.549876 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" podUID="df608734-a9f8-40c6-9430-1b25fa3389e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 31 17:35:21.101068 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:21.101034 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:21.101488 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:21.101357 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:21.102160 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:21.102138 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" Oct 31 17:35:21.157438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:35:21.157382 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" podStartSLOduration=9.157367037 podStartE2EDuration="9.157367037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:35:13.701780125 +0000 UTC m=+435.449754593" watchObservedRunningTime="2026-10-31 17:35:21.157367037 +0000 UTC m=+442.905341504" Oct 31 17:35:38.571280 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:35:38.571222 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:36:08.572371 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:36:08.572332 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:36:38.572116 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:36:38.572068 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:36:40.990710 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:40.990677 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:36:43.671307 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:43.671264 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr"] Oct 31 17:36:43.675865 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:43.675842 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" Oct 31 17:36:43.694407 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:43.694375 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr"] Oct 31 17:36:43.701694 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:43.701661 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-0e59ca\"" Oct 31 17:36:43.701895 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:43.701722 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-0e59ca-dockercfg-ppc99\"" Oct 31 17:36:43.760453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:43.760415 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjf7\" (UniqueName: \"kubernetes.io/projected/8c01da1b-48fc-49e4-9341-54965a5c45a0-kube-api-access-drjf7\") pod \"payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr\" (UID: \"8c01da1b-48fc-49e4-9341-54965a5c45a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" Oct 31 17:36:43.760640 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:43.760611 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c01da1b-48fc-49e4-9341-54965a5c45a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr\" (UID: \"8c01da1b-48fc-49e4-9341-54965a5c45a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" Oct 31 17:36:43.861657 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:43.861622 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjf7\" (UniqueName: \"kubernetes.io/projected/8c01da1b-48fc-49e4-9341-54965a5c45a0-kube-api-access-drjf7\") pod \"payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr\" (UID: \"8c01da1b-48fc-49e4-9341-54965a5c45a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" Oct 31 17:36:43.861857 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:43.861712 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c01da1b-48fc-49e4-9341-54965a5c45a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr\" (UID: \"8c01da1b-48fc-49e4-9341-54965a5c45a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" Oct 31 17:36:43.862484 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:43.862457 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c01da1b-48fc-49e4-9341-54965a5c45a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr\" (UID: \"8c01da1b-48fc-49e4-9341-54965a5c45a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" Oct 31 17:36:43.880325 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:43.880293 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjf7\" (UniqueName: \"kubernetes.io/projected/8c01da1b-48fc-49e4-9341-54965a5c45a0-kube-api-access-drjf7\") pod \"payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr\" (UID: \"8c01da1b-48fc-49e4-9341-54965a5c45a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" Oct 31 17:36:43.990146 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:43.990070 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" Oct 31 17:36:44.291975 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:44.291853 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr"] Oct 31 17:36:44.296972 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:36:44.296937 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c01da1b_48fc_49e4_9341_54965a5c45a0.slice/crio-99532bf855a201498e039fb1cfbdbca29d3ab63dc53e4c13fa9a03ccb88093db WatchSource:0}: Error finding container 99532bf855a201498e039fb1cfbdbca29d3ab63dc53e4c13fa9a03ccb88093db: Status 404 returned error can't find the container with id 99532bf855a201498e039fb1cfbdbca29d3ab63dc53e4c13fa9a03ccb88093db Oct 31 17:36:44.298988 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:44.298968 7618 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 17:36:45.114502 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:45.114470 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" event={"ID":"8c01da1b-48fc-49e4-9341-54965a5c45a0","Type":"ContainerStarted","Data":"99532bf855a201498e039fb1cfbdbca29d3ab63dc53e4c13fa9a03ccb88093db"} Oct 31 17:36:45.859911 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:45.859155 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:36:47.128140 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.128101 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" event={"ID":"8c01da1b-48fc-49e4-9341-54965a5c45a0","Type":"ContainerStarted","Data":"878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767"} Oct 31 17:36:47.128640 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.128169 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" Oct 31 17:36:47.164587 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.164535 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" podStartSLOduration=2.145815801 podStartE2EDuration="4.164518821s" totalImagesPullingTime="2.01870302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:36:47.16339488 +0000 UTC m=+528.911369347" watchObservedRunningTime="2026-10-31 17:36:47.164518821 +0000 UTC m=+528.912493342" Oct 31 17:36:47.635952 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.635918 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp"] Oct 31 17:36:47.640231 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.640203 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:47.667734 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.667705 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 17:36:47.667927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.667768 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 17:36:47.667927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.667852 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ee816e-serving-cert\"" Oct 31 17:36:47.667927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.667705 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 17:36:47.668099 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.667941 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4s8xz\"" Oct 31 17:36:47.670632 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.670140 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp"] Oct 31 17:36:47.702342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.702311 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fd9edb-9e10-4cac-8b27-a036c2718fce-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:47.702342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.702348 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-metrics-certs\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:47.702592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.702386 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkbr\" (UniqueName: \"kubernetes.io/projected/00fd9edb-9e10-4cac-8b27-a036c2718fce-kube-api-access-zgkbr\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:47.702592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.702531 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:47.804044 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.804010 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkbr\" (UniqueName: \"kubernetes.io/projected/00fd9edb-9e10-4cac-8b27-a036c2718fce-kube-api-access-zgkbr\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:47.804247 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.804072 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:47.804247 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.804158 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fd9edb-9e10-4cac-8b27-a036c2718fce-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:47.804247 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.804186 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-metrics-certs\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:47.804450 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:36:47.804248 7618 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-serving-cert: secret "maas-api-e2e-worker-w0-ee816e-serving-cert" not found Oct 31 17:36:47.804450 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:36:47.804352 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls podName:00fd9edb-9e10-4cac-8b27-a036c2718fce nodeName:}" failed. No retries permitted until 2026-10-31 17:36:48.304329185 +0000 UTC m=+530.052303651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls") pod "maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" (UID: "00fd9edb-9e10-4cac-8b27-a036c2718fce") : secret "maas-api-e2e-worker-w0-ee816e-serving-cert" not found Oct 31 17:36:47.804970 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.804944 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fd9edb-9e10-4cac-8b27-a036c2718fce-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:47.806934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.806906 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-metrics-certs\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:47.821652 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:47.821612 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkbr\" (UniqueName: \"kubernetes.io/projected/00fd9edb-9e10-4cac-8b27-a036c2718fce-kube-api-access-zgkbr\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:48.309234 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:48.309193 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:48.318142 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:48.313044 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:48.553794 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:48.553763 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:48.851580 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:48.851493 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp"] Oct 31 17:36:48.855679 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:36:48.855651 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fd9edb_9e10_4cac_8b27_a036c2718fce.slice/crio-1005cf2d6e5299a1c681956cbd002004fed5e2fcf885d2cc8c85407f44174a7c WatchSource:0}: Error finding container 1005cf2d6e5299a1c681956cbd002004fed5e2fcf885d2cc8c85407f44174a7c: Status 404 returned error can't find the container with id 1005cf2d6e5299a1c681956cbd002004fed5e2fcf885d2cc8c85407f44174a7c Oct 31 17:36:49.142323 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:49.142285 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" event={"ID":"00fd9edb-9e10-4cac-8b27-a036c2718fce","Type":"ContainerStarted","Data":"1005cf2d6e5299a1c681956cbd002004fed5e2fcf885d2cc8c85407f44174a7c"} Oct 31 17:36:51.154846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:51.154802 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" event={"ID":"00fd9edb-9e10-4cac-8b27-a036c2718fce","Type":"ContainerStarted","Data":"ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02"} Oct 31 17:36:51.155381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:51.154924 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:51.188556 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:51.188498 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" podStartSLOduration=2.743430634 podStartE2EDuration="4.188482453s" totalImagesPullingTime="1.445051819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:36:51.187916429 +0000 UTC m=+532.935890897" watchObservedRunningTime="2026-10-31 17:36:51.188482453 +0000 UTC m=+532.936456920" Oct 31 17:36:51.354708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:51.354666 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:36:56.637948 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.637914 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q"] Oct 31 17:36:56.642039 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.642017 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.659370 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.659340 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d80080\"/\"kube-root-ca.crt\"" Oct 31 17:36:56.666703 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.666671 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d80080\"/\"simulator-w6-d80080-kserve-self-signed-certs\"" Oct 31 17:36:56.666899 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.666776 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d80080\"/\"openshift-service-ca.crt\"" Oct 31 17:36:56.669171 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.669139 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q"] Oct 31 17:36:56.669628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.669608 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d80080\"/\"default-dockercfg-7tv7m\"" Oct 31 17:36:56.787151 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.787118 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:36:56.802137 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.802095 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-dshm\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.802341 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.802163 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrfbk\" (UniqueName: \"kubernetes.io/projected/67be76a3-2fc6-4e3e-8690-7f6549760cea-kube-api-access-nrfbk\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.802341 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.802225 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67be76a3-2fc6-4e3e-8690-7f6549760cea-tls-certs\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.802341 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.802287 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-model-cache\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.802463 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.802368 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-home\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.802463 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.802432 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-tmp-dir\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.903726 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.903691 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-dshm\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.903922 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.903765 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrfbk\" (UniqueName: \"kubernetes.io/projected/67be76a3-2fc6-4e3e-8690-7f6549760cea-kube-api-access-nrfbk\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.903922 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.903791 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67be76a3-2fc6-4e3e-8690-7f6549760cea-tls-certs\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.903922 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.903829 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-model-cache\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.903922 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.903855 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-home\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.903922 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.903913 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-tmp-dir\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.904546 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.904425 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-model-cache\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.904546 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.904537 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-home\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.904840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.904816 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-tmp-dir\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.907197 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.907166 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-dshm\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.907496 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.907445 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67be76a3-2fc6-4e3e-8690-7f6549760cea-tls-certs\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.920969 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.920937 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrfbk\" (UniqueName: \"kubernetes.io/projected/67be76a3-2fc6-4e3e-8690-7f6549760cea-kube-api-access-nrfbk\") pod \"simulator-w6-d80080-kserve-c7885fcb5-kmm7q\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:56.957862 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:56.957822 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:36:57.167855 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:57.167778 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:36:57.259783 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:57.259745 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q"] Oct 31 17:36:57.265365 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:36:57.265335 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67be76a3_2fc6_4e3e_8690_7f6549760cea.slice/crio-8a4071e76d5b31e20a533f68442b3755e8aa46ab99585987ae7d1e1816ac6139 WatchSource:0}: Error finding container 8a4071e76d5b31e20a533f68442b3755e8aa46ab99585987ae7d1e1816ac6139: Status 404 returned error can't find the container with id 8a4071e76d5b31e20a533f68442b3755e8aa46ab99585987ae7d1e1816ac6139 Oct 31 17:36:58.139406 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:58.139377 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" Oct 31 17:36:58.196929 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:58.196881 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" event={"ID":"67be76a3-2fc6-4e3e-8690-7f6549760cea","Type":"ContainerStarted","Data":"8a4071e76d5b31e20a533f68442b3755e8aa46ab99585987ae7d1e1816ac6139"} Oct 31 17:36:59.204335 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:59.204292 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" event={"ID":"67be76a3-2fc6-4e3e-8690-7f6549760cea","Type":"ContainerStarted","Data":"c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3"} Oct 31 17:36:59.238076 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:59.238018 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podStartSLOduration=1.923031269 podStartE2EDuration="3.238003641s" totalImagesPullingTime="1.314972372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:36:59.237171588 +0000 UTC m=+540.985146057" watchObservedRunningTime="2026-10-31 17:36:59.238003641 +0000 UTC m=+540.985978108" Oct 31 17:36:59.958923 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:36:59.958879 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:37:04.486382 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:04.486347 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:37:06.280465 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.280303 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5"] Oct 31 17:37:06.284232 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.284209 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.300417 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.300389 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cf0a86\"/\"default-dockercfg-94drc\"" Oct 31 17:37:06.307978 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.307945 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cf0a86\"/\"simulator-w3-cf0a86-kserve-self-signed-certs\"" Oct 31 17:37:06.321366 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.321333 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5"] Oct 31 17:37:06.382033 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.382006 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cf0a86\"/\"openshift-service-ca.crt\"" Oct 31 17:37:06.382205 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.382118 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cf0a86\"/\"kube-root-ca.crt\"" Oct 31 17:37:06.409342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.409310 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-home\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.409542 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.409394 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tmp-dir\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.409542 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.409514 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn8vs\" (UniqueName: \"kubernetes.io/projected/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-kube-api-access-fn8vs\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.409656 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.409562 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-dshm\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.409711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.409673 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-model-cache\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.409711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.409691 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tls-certs\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.510563 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.510513 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-dshm\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.510780 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.510616 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-model-cache\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.510780 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.510642 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tls-certs\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.510780 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.510694 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-home\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.510953 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.510801 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tmp-dir\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.510953 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.510850 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn8vs\" (UniqueName: \"kubernetes.io/projected/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-kube-api-access-fn8vs\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.511125 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.511075 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-home\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.511201 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.511091 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-model-cache\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.511351 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.511322 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tmp-dir\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.513740 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.513713 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-dshm\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.514914 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.514890 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tls-certs\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.531811 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.531754 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn8vs\" (UniqueName: \"kubernetes.io/projected/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-kube-api-access-fn8vs\") pod \"simulator-w3-cf0a86-kserve-549d995b6b-xzql5\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.601043 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.601007 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:06.903717 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.903677 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5"] Oct 31 17:37:06.906873 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:37:06.906840 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb7376f2_8c9a_4f09_89f8_fd8ee88e80c6.slice/crio-2b831e8d32f72d28cad1665e1cb849572cf8898654966316d7f74d315659ab79 WatchSource:0}: Error finding container 2b831e8d32f72d28cad1665e1cb849572cf8898654966316d7f74d315659ab79: Status 404 returned error can't find the container with id 2b831e8d32f72d28cad1665e1cb849572cf8898654966316d7f74d315659ab79 Oct 31 17:37:06.958052 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.958023 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:37:06.971857 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:06.971833 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:37:07.252042 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:07.251955 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" event={"ID":"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6","Type":"ContainerStarted","Data":"91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137"} Oct 31 17:37:07.252042 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:07.252003 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" event={"ID":"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6","Type":"ContainerStarted","Data":"2b831e8d32f72d28cad1665e1cb849572cf8898654966316d7f74d315659ab79"} Oct 31 17:37:07.263766 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:07.263711 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:37:07.287170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:07.287116 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podStartSLOduration=1.287099111 podStartE2EDuration="1.287099111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:37:07.285199915 +0000 UTC m=+549.033174395" watchObservedRunningTime="2026-10-31 17:37:07.287099111 +0000 UTC m=+549.035073579" Oct 31 17:37:07.601229 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:07.601140 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:08.575785 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:37:08.575744 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:37:12.145896 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.145807 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl"] Oct 31 17:37:12.150248 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.150221 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.167818 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.167791 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl"] Oct 31 17:37:12.176341 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.176310 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1c0bfd\"/\"openshift-service-ca.crt\"" Oct 31 17:37:12.176529 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.176352 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1c0bfd\"/\"default-dockercfg-x4sxr\"" Oct 31 17:37:12.176529 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.176374 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1c0bfd\"/\"simulator-w4-1c0bfd-kserve-self-signed-certs\"" Oct 31 17:37:12.176529 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.176428 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1c0bfd\"/\"kube-root-ca.crt\"" Oct 31 17:37:12.276424 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.276389 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-model-cache\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.276424 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.276439 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-home\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.276714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.276556 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-dshm\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.276714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.276612 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tls-certs\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.276714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.276702 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q78j7\" (UniqueName: \"kubernetes.io/projected/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-kube-api-access-q78j7\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.276878 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.276735 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tmp-dir\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.377413 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.377364 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q78j7\" (UniqueName: \"kubernetes.io/projected/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-kube-api-access-q78j7\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.377619 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.377438 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tmp-dir\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.377619 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.377532 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-model-cache\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.377619 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.377577 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-home\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.377793 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.377636 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-dshm\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.377793 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.377693 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tls-certs\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.377901 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.377819 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tmp-dir\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.377955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.377940 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-model-cache\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.378029 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.378005 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-home\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.379940 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.379910 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-dshm\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.380203 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.380182 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tls-certs\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.393940 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.393916 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q78j7\" (UniqueName: \"kubernetes.io/projected/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-kube-api-access-q78j7\") pod \"simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.464096 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.464003 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:12.760496 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:12.760402 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl"] Oct 31 17:37:12.764464 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:37:12.764434 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4f22d3d_03f9_495b_9b3e_bbc3aea680fd.slice/crio-273266ef794934e658c79cff71e4502abb5fdc82ae822d576a755132a97362ca WatchSource:0}: Error finding container 273266ef794934e658c79cff71e4502abb5fdc82ae822d576a755132a97362ca: Status 404 returned error can't find the container with id 273266ef794934e658c79cff71e4502abb5fdc82ae822d576a755132a97362ca Oct 31 17:37:13.291492 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:13.291449 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" event={"ID":"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd","Type":"ContainerStarted","Data":"d5f5747c0474a8e49f264dd8bcddee1ad1a6b7c813c000536681d92469559eda"} Oct 31 17:37:13.291900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:13.291503 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" event={"ID":"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd","Type":"ContainerStarted","Data":"273266ef794934e658c79cff71e4502abb5fdc82ae822d576a755132a97362ca"} Oct 31 17:37:13.326863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:13.326799 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podStartSLOduration=1.326737664 podStartE2EDuration="1.326737664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:37:13.325192272 +0000 UTC m=+555.073166739" watchObservedRunningTime="2026-10-31 17:37:13.326737664 +0000 UTC m=+555.074712131" Oct 31 17:37:13.464440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:13.464403 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:15.691188 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:15.691147 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:37:16.602191 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:16.602155 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:16.617025 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:16.616993 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:17.342144 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:17.342108 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:37:20.652778 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.652741 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk"] Oct 31 17:37:20.703650 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.703611 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk"] Oct 31 17:37:20.703847 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.703735 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.719119 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.719068 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5c446c\"/\"kube-root-ca.crt\"" Oct 31 17:37:20.719292 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.719132 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5c446c\"/\"simulator-w2-5c446c-kserve-self-signed-certs\"" Oct 31 17:37:20.729660 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.729621 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5c446c\"/\"default-dockercfg-52ml5\"" Oct 31 17:37:20.729660 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.729642 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5c446c\"/\"openshift-service-ca.crt\"" Oct 31 17:37:20.779067 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.779032 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tmp-dir\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.779245 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.779096 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-home\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.779245 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.779200 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpmlz\" (UniqueName: \"kubernetes.io/projected/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-kube-api-access-cpmlz\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.779419 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.779282 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tls-certs\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.779419 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.779348 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-dshm\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.779536 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.779477 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-model-cache\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.880179 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.880134 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-model-cache\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.880179 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.880186 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tmp-dir\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.880529 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.880217 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-home\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.880529 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.880239 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmlz\" (UniqueName: \"kubernetes.io/projected/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-kube-api-access-cpmlz\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.880529 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.880298 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tls-certs\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.880529 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.880351 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-dshm\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.880774 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.880691 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-model-cache\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.880774 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.880765 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-home\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.880874 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.880819 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tmp-dir\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.882558 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.882538 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-dshm\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.883032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.883008 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tls-certs\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:20.901673 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:20.901635 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmlz\" (UniqueName: \"kubernetes.io/projected/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-kube-api-access-cpmlz\") pod \"simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:21.034716 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:21.034616 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:21.352269 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:21.352145 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk"] Oct 31 17:37:22.373569 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:22.373532 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" event={"ID":"4e3c66c5-efb3-48a3-830b-bae3b90d74f5","Type":"ContainerStarted","Data":"23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0"} Oct 31 17:37:22.373935 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:22.373577 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" event={"ID":"4e3c66c5-efb3-48a3-830b-bae3b90d74f5","Type":"ContainerStarted","Data":"aba84e853f932c3d7f7e8cf82a6ea26f263e80da49b4e3ba24848e0cbca68e2c"} Oct 31 17:37:22.403967 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:22.403902 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podStartSLOduration=2.403884219 podStartE2EDuration="2.403884219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:37:22.402528229 +0000 UTC m=+564.150502697" watchObservedRunningTime="2026-10-31 17:37:22.403884219 +0000 UTC m=+564.151858690" Oct 31 17:37:22.465032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:22.464994 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:22.477774 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:22.477742 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:23.035398 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:23.035357 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:23.387371 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:23.387287 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:37:25.669668 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:25.669627 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:37:31.035612 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:31.035565 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:31.048996 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:31.048966 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:31.432438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:31.432408 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:37:33.286153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.286109 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr"] Oct 31 17:37:33.291935 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.291897 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.315265 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.315206 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d80080\"/\"premium-w6-d80080-kserve-self-signed-certs\"" Oct 31 17:37:33.317093 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.317070 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr"] Oct 31 17:37:33.319703 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.319675 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdhls\" (UniqueName: \"kubernetes.io/projected/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-kube-api-access-fdhls\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.319818 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.319734 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tls-certs\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.319863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.319818 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tmp-dir\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.319955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.319931 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-home\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.320014 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.319997 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-dshm\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.320079 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.320067 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-model-cache\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.420783 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.420736 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhls\" (UniqueName: \"kubernetes.io/projected/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-kube-api-access-fdhls\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.420997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.420813 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tls-certs\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.420997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.420846 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tmp-dir\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.420997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.420885 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-home\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.420997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.420933 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-dshm\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.421224 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.421007 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-model-cache\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.421354 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.421329 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tmp-dir\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.421485 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.421460 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-home\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.421814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.421785 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-model-cache\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.423880 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.423853 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-dshm\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.424500 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.424477 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tls-certs\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.440460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.440431 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhls\" (UniqueName: \"kubernetes.io/projected/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-kube-api-access-fdhls\") pod \"premium-w6-d80080-kserve-675dc985dd-l2zdr\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.605865 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.605773 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:33.904150 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:33.904107 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr"] Oct 31 17:37:33.908234 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:37:33.908200 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfbf57d8_cd64_45ae_9cc0_7b65a52207b5.slice/crio-0d36c104ae65c93e50327495cf1616a4ae7c3573f94d63eca8423797f9c0ebd7 WatchSource:0}: Error finding container 0d36c104ae65c93e50327495cf1616a4ae7c3573f94d63eca8423797f9c0ebd7: Status 404 returned error can't find the container with id 0d36c104ae65c93e50327495cf1616a4ae7c3573f94d63eca8423797f9c0ebd7 Oct 31 17:37:34.442449 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:34.442405 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" event={"ID":"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5","Type":"ContainerStarted","Data":"efac4f278c3a2021b406ac0396922eb5b265fbd1b172fa687c00e9f34643fb2b"} Oct 31 17:37:34.442887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:34.442463 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" event={"ID":"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5","Type":"ContainerStarted","Data":"0d36c104ae65c93e50327495cf1616a4ae7c3573f94d63eca8423797f9c0ebd7"} Oct 31 17:37:34.477726 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:34.477666 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podStartSLOduration=1.4776515510000001 podStartE2EDuration="1.477651551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:37:34.476233058 +0000 UTC m=+576.224207537" watchObservedRunningTime="2026-10-31 17:37:34.477651551 +0000 UTC m=+576.225626018" Oct 31 17:37:34.606463 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:34.606422 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:38.574315 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:37:38.574275 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:37:39.875669 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:39.875630 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:37:43.606601 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:43.606557 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:43.620121 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:43.620091 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:44.510532 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:44.510497 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:37:49.165512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.165474 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96"] Oct 31 17:37:49.171049 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.171004 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.194926 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.194896 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cf0a86\"/\"premium-w3-cf0a86-kserve-self-signed-certs\"" Oct 31 17:37:49.196893 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.196864 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96"] Oct 31 17:37:49.293520 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.293472 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-model-cache\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.293728 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.293535 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54q2s\" (UniqueName: \"kubernetes.io/projected/d3b13cb1-a3f6-4421-b295-547e34fd90be-kube-api-access-54q2s\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.293728 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.293710 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-tmp-dir\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.293846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.293750 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-dshm\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.293898 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.293865 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b13cb1-a3f6-4421-b295-547e34fd90be-tls-certs\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.293955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.293896 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-home\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.394827 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.394778 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b13cb1-a3f6-4421-b295-547e34fd90be-tls-certs\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.395045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.394839 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-home\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.395045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.394914 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-model-cache\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.395045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.394948 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54q2s\" (UniqueName: \"kubernetes.io/projected/d3b13cb1-a3f6-4421-b295-547e34fd90be-kube-api-access-54q2s\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.395045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.395012 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-tmp-dir\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.395045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.395042 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-dshm\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.395366 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.395288 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-home\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.395644 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.395615 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-model-cache\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.395808 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.395788 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-tmp-dir\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.397778 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.397727 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-dshm\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.403420 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.403394 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b13cb1-a3f6-4421-b295-547e34fd90be-tls-certs\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.412296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.412273 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54q2s\" (UniqueName: \"kubernetes.io/projected/d3b13cb1-a3f6-4421-b295-547e34fd90be-kube-api-access-54q2s\") pod \"premium-w3-cf0a86-kserve-795967bb4c-jrw96\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.486292 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.486186 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:49.788232 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:49.788146 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96"] Oct 31 17:37:49.792629 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:37:49.792597 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3b13cb1_a3f6_4421_b295_547e34fd90be.slice/crio-43afb1b50300aa8377716df219e7e2b42c87737824095ecb1f56eaf29b75bf5e WatchSource:0}: Error finding container 43afb1b50300aa8377716df219e7e2b42c87737824095ecb1f56eaf29b75bf5e: Status 404 returned error can't find the container with id 43afb1b50300aa8377716df219e7e2b42c87737824095ecb1f56eaf29b75bf5e Oct 31 17:37:50.556840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:50.556794 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" event={"ID":"d3b13cb1-a3f6-4421-b295-547e34fd90be","Type":"ContainerStarted","Data":"9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0"} Oct 31 17:37:50.556840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:50.556839 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" event={"ID":"d3b13cb1-a3f6-4421-b295-547e34fd90be","Type":"ContainerStarted","Data":"43afb1b50300aa8377716df219e7e2b42c87737824095ecb1f56eaf29b75bf5e"} Oct 31 17:37:50.591836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:50.591783 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podStartSLOduration=1.5917682960000001 podStartE2EDuration="1.591768296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:37:50.590562015 +0000 UTC m=+592.338536483" watchObservedRunningTime="2026-10-31 17:37:50.591768296 +0000 UTC m=+592.339742792" Oct 31 17:37:51.486423 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:51.486379 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:55.148565 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:55.148526 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:37:58.467214 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:58.467178 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:37:58.468385 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:58.468361 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:37:59.486622 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:59.486583 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:59.499429 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:59.499398 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:37:59.627116 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:37:59.627084 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:38:01.470515 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.470482 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5"] Oct 31 17:38:01.475665 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.475646 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.493721 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.493692 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5"] Oct 31 17:38:01.501515 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.501487 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b78c31\"/\"openshift-service-ca.crt\"" Oct 31 17:38:01.501765 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.501606 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b78c31\"/\"kube-root-ca.crt\"" Oct 31 17:38:01.501765 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.501745 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b78c31\"/\"premium-w5-b78c31-kserve-self-signed-certs\"" Oct 31 17:38:01.501765 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.501622 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b78c31\"/\"default-dockercfg-9mrcp\"" Oct 31 17:38:01.635048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.632902 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-dshm\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.635048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.633001 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-model-cache\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.635048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.633051 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5803fb07-04c1-4d98-8278-b0f21077d21b-tls-certs\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.635048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.633084 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-tmp-dir\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.635048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.633113 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-home\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.635048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.633202 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r5bj\" (UniqueName: \"kubernetes.io/projected/5803fb07-04c1-4d98-8278-b0f21077d21b-kube-api-access-6r5bj\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.733871 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.733772 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-model-cache\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.733871 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.733832 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5803fb07-04c1-4d98-8278-b0f21077d21b-tls-certs\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.734122 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.733874 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-tmp-dir\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.734122 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.733910 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-home\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.734122 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.734000 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5bj\" (UniqueName: \"kubernetes.io/projected/5803fb07-04c1-4d98-8278-b0f21077d21b-kube-api-access-6r5bj\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.734122 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.734088 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-dshm\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.734457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.734429 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-tmp-dir\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.734666 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.734647 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-model-cache\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.734748 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.734712 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-home\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.736776 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.736749 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-dshm\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.737013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.736995 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5803fb07-04c1-4d98-8278-b0f21077d21b-tls-certs\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.751891 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.751790 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5bj\" (UniqueName: \"kubernetes.io/projected/5803fb07-04c1-4d98-8278-b0f21077d21b-kube-api-access-6r5bj\") pod \"premium-w5-b78c31-kserve-74d5fb78c-l85h5\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:01.789566 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:01.789531 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:02.092336 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:02.092238 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5"] Oct 31 17:38:02.096923 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:38:02.096885 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5803fb07_04c1_4d98_8278_b0f21077d21b.slice/crio-2a024f7488260bbc569de4e84e249945957f575fa91eeb73b3ac2a6f1ab70bd2 WatchSource:0}: Error finding container 2a024f7488260bbc569de4e84e249945957f575fa91eeb73b3ac2a6f1ab70bd2: Status 404 returned error can't find the container with id 2a024f7488260bbc569de4e84e249945957f575fa91eeb73b3ac2a6f1ab70bd2 Oct 31 17:38:02.648109 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:02.648061 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" event={"ID":"5803fb07-04c1-4d98-8278-b0f21077d21b","Type":"ContainerStarted","Data":"75da3d280119c73a000df361198ad8877d13506b2b82195e76f1414210349bf4"} Oct 31 17:38:02.648597 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:02.648120 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" event={"ID":"5803fb07-04c1-4d98-8278-b0f21077d21b","Type":"ContainerStarted","Data":"2a024f7488260bbc569de4e84e249945957f575fa91eeb73b3ac2a6f1ab70bd2"} Oct 31 17:38:02.684658 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:02.684605 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podStartSLOduration=1.684590947 podStartE2EDuration="1.684590947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:38:02.683627604 +0000 UTC m=+604.431602073" watchObservedRunningTime="2026-10-31 17:38:02.684590947 +0000 UTC m=+604.432565414" Oct 31 17:38:02.790488 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:02.790439 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:08.572851 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:38:08.572815 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:38:11.789924 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:11.789881 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:11.802801 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:11.802763 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:12.716233 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:12.716199 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:38:13.887546 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:13.887508 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:38:34.256382 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.256348 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl"] Oct 31 17:38:34.261933 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.261907 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:34.286430 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.286397 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:38:34.288932 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.288904 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl"] Oct 31 17:38:34.379038 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.379009 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:38:34.383565 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.383543 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3250a0f-qr9hl\" (UID: \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:34.383650 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.383581 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7q22\" (UniqueName: \"kubernetes.io/projected/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-kube-api-access-q7q22\") pod \"maas-api-revoke-sub-e3250a0f-qr9hl\" (UID: \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:34.383709 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.383660 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3250a0f-qr9hl\" (UID: \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:34.484628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.484576 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3250a0f-qr9hl\" (UID: \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:34.484844 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.484637 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7q22\" (UniqueName: \"kubernetes.io/projected/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-kube-api-access-q7q22\") pod \"maas-api-revoke-sub-e3250a0f-qr9hl\" (UID: \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:34.484844 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.484683 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3250a0f-qr9hl\" (UID: \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:34.485450 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.485420 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3250a0f-qr9hl\" (UID: \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:34.487106 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.487081 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3250a0f-qr9hl\" (UID: \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:34.501691 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.501663 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7q22\" (UniqueName: \"kubernetes.io/projected/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-kube-api-access-q7q22\") pod \"maas-api-revoke-sub-e3250a0f-qr9hl\" (UID: \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:34.578663 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.578567 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:34.879132 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:34.879054 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl"] Oct 31 17:38:34.882674 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:38:34.882643 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5aed79_4d7b_4c30_b958_afac9cbe69d6.slice/crio-3f1b8169edd9964a80fec7cccb11feb92c0f89a0156535488612a8e875694a6b WatchSource:0}: Error finding container 3f1b8169edd9964a80fec7cccb11feb92c0f89a0156535488612a8e875694a6b: Status 404 returned error can't find the container with id 3f1b8169edd9964a80fec7cccb11feb92c0f89a0156535488612a8e875694a6b Oct 31 17:38:35.838389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:35.838347 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" event={"ID":"4c5aed79-4d7b-4c30-b958-afac9cbe69d6","Type":"ContainerStarted","Data":"3f1b8169edd9964a80fec7cccb11feb92c0f89a0156535488612a8e875694a6b"} Oct 31 17:38:35.910841 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:35.910804 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl"] Oct 31 17:38:35.914667 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:35.914640 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:35.932635 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:35.932603 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl"] Oct 31 17:38:36.003802 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:36.003759 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwd7c\" (UniqueName: \"kubernetes.io/projected/53a289ed-e730-4e17-a65c-0d2fc1b5593c-kube-api-access-fwd7c\") pod \"maas-api-revoke-sub-7c8c58c0-8v8sl\" (UID: \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:36.003993 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:36.003849 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c8c58c0-8v8sl\" (UID: \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:36.003993 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:36.003962 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c8c58c0-8v8sl\" (UID: \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:36.105189 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:36.105108 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwd7c\" (UniqueName: \"kubernetes.io/projected/53a289ed-e730-4e17-a65c-0d2fc1b5593c-kube-api-access-fwd7c\") pod \"maas-api-revoke-sub-7c8c58c0-8v8sl\" (UID: \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:36.105189 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:36.105183 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c8c58c0-8v8sl\" (UID: \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:36.108410 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:36.105234 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c8c58c0-8v8sl\" (UID: \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:36.108410 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:36.105934 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c8c58c0-8v8sl\" (UID: \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:36.108410 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:36.107860 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c8c58c0-8v8sl\" (UID: \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:36.123630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:36.123601 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwd7c\" (UniqueName: \"kubernetes.io/projected/53a289ed-e730-4e17-a65c-0d2fc1b5593c-kube-api-access-fwd7c\") pod \"maas-api-revoke-sub-7c8c58c0-8v8sl\" (UID: \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:36.246053 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:36.246021 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:36.555401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:36.555362 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl"] Oct 31 17:38:36.559610 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:38:36.559574 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a289ed_e730_4e17_a65c_0d2fc1b5593c.slice/crio-b414e8a5a1e09f7c026ed8ad3bd5a31b4f0e9a4b96b5cab18108cd0572cf4995 WatchSource:0}: Error finding container b414e8a5a1e09f7c026ed8ad3bd5a31b4f0e9a4b96b5cab18108cd0572cf4995: Status 404 returned error can't find the container with id b414e8a5a1e09f7c026ed8ad3bd5a31b4f0e9a4b96b5cab18108cd0572cf4995 Oct 31 17:38:36.849732 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:36.848814 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" event={"ID":"53a289ed-e730-4e17-a65c-0d2fc1b5593c","Type":"ContainerStarted","Data":"b414e8a5a1e09f7c026ed8ad3bd5a31b4f0e9a4b96b5cab18108cd0572cf4995"} Oct 31 17:38:37.857225 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:37.857186 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" event={"ID":"4c5aed79-4d7b-4c30-b958-afac9cbe69d6","Type":"ContainerStarted","Data":"b727cfcf7a1960be58d1f6cfb473091cc7f874dd20b098a8c91117d309af8161"} Oct 31 17:38:37.891202 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:37.891151 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" podStartSLOduration=1.4194807950000001 podStartE2EDuration="3.891137274s" totalImagesPullingTime="2.471656479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:38:37.890966941 +0000 UTC m=+639.638941390" watchObservedRunningTime="2026-10-31 17:38:37.891137274 +0000 UTC m=+639.639111741" Oct 31 17:38:38.581746 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:38:38.581650 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:38:38.865600 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:38.865496 7618 generic.go:356] "Generic (PLEG): container finished" podID="4c5aed79-4d7b-4c30-b958-afac9cbe69d6" containerID="b727cfcf7a1960be58d1f6cfb473091cc7f874dd20b098a8c91117d309af8161" exitCode=0 Oct 31 17:38:38.865600 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:38.865574 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" event={"ID":"4c5aed79-4d7b-4c30-b958-afac9cbe69d6","Type":"ContainerDied","Data":"b727cfcf7a1960be58d1f6cfb473091cc7f874dd20b098a8c91117d309af8161"} Oct 31 17:38:38.867466 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:38.867441 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" event={"ID":"53a289ed-e730-4e17-a65c-0d2fc1b5593c","Type":"ContainerStarted","Data":"8922d6b8a7e89ef825f36624de7d40fa5b7e5969ca1ef8ba49837cc2d233dc23"} Oct 31 17:38:38.935341 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:38.935277 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" podStartSLOduration=2.585920043 podStartE2EDuration="3.935241102s" totalImagesPullingTime="1.349321059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:38:38.934019043 +0000 UTC m=+640.681993510" watchObservedRunningTime="2026-10-31 17:38:38.935241102 +0000 UTC m=+640.683215572" Oct 31 17:38:39.874577 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:39.874492 7618 generic.go:356] "Generic (PLEG): container finished" podID="53a289ed-e730-4e17-a65c-0d2fc1b5593c" containerID="8922d6b8a7e89ef825f36624de7d40fa5b7e5969ca1ef8ba49837cc2d233dc23" exitCode=0 Oct 31 17:38:39.874577 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:39.874539 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" event={"ID":"53a289ed-e730-4e17-a65c-0d2fc1b5593c","Type":"ContainerDied","Data":"8922d6b8a7e89ef825f36624de7d40fa5b7e5969ca1ef8ba49837cc2d233dc23"} Oct 31 17:38:40.034734 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.034702 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:40.153593 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.153541 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-openshift-ca\") pod \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\" (UID: \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\") " Oct 31 17:38:40.153821 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.153687 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-kube-api-access-q7q22\" (UniqueName: \"kubernetes.io/projected/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-kube-api-access-q7q22\") pod \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\" (UID: \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\") " Oct 31 17:38:40.153821 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.153725 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-xks-ca\") pod \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\" (UID: \"4c5aed79-4d7b-4c30-b958-afac9cbe69d6\") " Oct 31 17:38:40.153955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.153929 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-openshift-ca" pod "4c5aed79-4d7b-4c30-b958-afac9cbe69d6" (UID: "4c5aed79-4d7b-4c30-b958-afac9cbe69d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:38:40.154121 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.154105 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:40.156020 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.155990 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-xks-ca" pod "4c5aed79-4d7b-4c30-b958-afac9cbe69d6" (UID: "4c5aed79-4d7b-4c30-b958-afac9cbe69d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:38:40.156169 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.156026 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-kube-api-access-q7q22" pod "4c5aed79-4d7b-4c30-b958-afac9cbe69d6" (UID: "4c5aed79-4d7b-4c30-b958-afac9cbe69d6"). InnerVolumeSpecName "kube-api-access-q7q22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:38:40.255775 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.255736 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7q22\" (UniqueName: \"kubernetes.io/projected/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-kube-api-access-q7q22\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:40.255775 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.255777 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5aed79-4d7b-4c30-b958-afac9cbe69d6-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:40.880920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.880888 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" Oct 31 17:38:40.881367 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.880922 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl" event={"ID":"4c5aed79-4d7b-4c30-b958-afac9cbe69d6","Type":"ContainerDied","Data":"3f1b8169edd9964a80fec7cccb11feb92c0f89a0156535488612a8e875694a6b"} Oct 31 17:38:40.881367 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:40.880979 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f1b8169edd9964a80fec7cccb11feb92c0f89a0156535488612a8e875694a6b" Oct 31 17:38:41.033677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.033642 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:41.166691 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.166658 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a289ed-e730-4e17-a65c-0d2fc1b5593c-kube-api-access-fwd7c\" (UniqueName: \"kubernetes.io/projected/53a289ed-e730-4e17-a65c-0d2fc1b5593c-kube-api-access-fwd7c\") pod \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\" (UID: \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\") " Oct 31 17:38:41.166897 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.166818 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-xks-ca\") pod \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\" (UID: \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\") " Oct 31 17:38:41.166897 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.166875 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-openshift-ca\") pod \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\" (UID: \"53a289ed-e730-4e17-a65c-0d2fc1b5593c\") " Oct 31 17:38:41.167294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.167245 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-openshift-ca" pod "53a289ed-e730-4e17-a65c-0d2fc1b5593c" (UID: "53a289ed-e730-4e17-a65c-0d2fc1b5593c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:38:41.168892 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.168863 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a289ed-e730-4e17-a65c-0d2fc1b5593c-kube-api-access-fwd7c" pod "53a289ed-e730-4e17-a65c-0d2fc1b5593c" (UID: "53a289ed-e730-4e17-a65c-0d2fc1b5593c"). InnerVolumeSpecName "kube-api-access-fwd7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:38:41.168892 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.168875 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-xks-ca" pod "53a289ed-e730-4e17-a65c-0d2fc1b5593c" (UID: "53a289ed-e730-4e17-a65c-0d2fc1b5593c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:38:41.268690 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.268608 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwd7c\" (UniqueName: \"kubernetes.io/projected/53a289ed-e730-4e17-a65c-0d2fc1b5593c-kube-api-access-fwd7c\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:41.268690 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.268639 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:41.268690 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.268650 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a289ed-e730-4e17-a65c-0d2fc1b5593c-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:41.545111 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.545020 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:38:41.890652 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.890624 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" Oct 31 17:38:41.891056 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.890641 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl" event={"ID":"53a289ed-e730-4e17-a65c-0d2fc1b5593c","Type":"ContainerDied","Data":"b414e8a5a1e09f7c026ed8ad3bd5a31b4f0e9a4b96b5cab18108cd0572cf4995"} Oct 31 17:38:41.891056 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:41.890704 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b414e8a5a1e09f7c026ed8ad3bd5a31b4f0e9a4b96b5cab18108cd0572cf4995" Oct 31 17:38:50.165786 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.165746 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh"] Oct 31 17:38:50.166504 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.166483 7618 cpu_manager.go:414] "containerMap: removing container" podUID="4c5aed79-4d7b-4c30-b958-afac9cbe69d6" containerName="revoke-keys" Oct 31 17:38:50.166555 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.166508 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5aed79-4d7b-4c30-b958-afac9cbe69d6" containerName="revoke-keys" Oct 31 17:38:50.166555 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.166540 7618 cpu_manager.go:414] "containerMap: removing container" podUID="53a289ed-e730-4e17-a65c-0d2fc1b5593c" containerName="revoke-keys" Oct 31 17:38:50.166555 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.166547 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a289ed-e730-4e17-a65c-0d2fc1b5593c" containerName="revoke-keys" Oct 31 17:38:50.166690 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.166677 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5aed79-4d7b-4c30-b958-afac9cbe69d6" containerName="revoke-keys" Oct 31 17:38:50.166724 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.166695 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a289ed-e730-4e17-a65c-0d2fc1b5593c" containerName="revoke-keys" Oct 31 17:38:50.170074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.170057 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:50.186003 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.185973 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh"] Oct 31 17:38:50.193432 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.193400 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:38:50.193432 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.193415 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:38:50.261338 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.261299 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlp7s\" (UniqueName: \"kubernetes.io/projected/b04d20be-e50b-432d-918e-db1854292fc2-kube-api-access-dlp7s\") pod \"maas-api-revoke-sub-ab459dfd-bvdgh\" (UID: \"b04d20be-e50b-432d-918e-db1854292fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:50.261536 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.261414 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab459dfd-bvdgh\" (UID: \"b04d20be-e50b-432d-918e-db1854292fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:50.261536 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.261446 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab459dfd-bvdgh\" (UID: \"b04d20be-e50b-432d-918e-db1854292fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:50.362644 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.362605 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlp7s\" (UniqueName: \"kubernetes.io/projected/b04d20be-e50b-432d-918e-db1854292fc2-kube-api-access-dlp7s\") pod \"maas-api-revoke-sub-ab459dfd-bvdgh\" (UID: \"b04d20be-e50b-432d-918e-db1854292fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:50.362906 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.362656 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab459dfd-bvdgh\" (UID: \"b04d20be-e50b-432d-918e-db1854292fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:50.362906 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.362676 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab459dfd-bvdgh\" (UID: \"b04d20be-e50b-432d-918e-db1854292fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:50.363356 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.363328 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab459dfd-bvdgh\" (UID: \"b04d20be-e50b-432d-918e-db1854292fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:50.365160 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.365134 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab459dfd-bvdgh\" (UID: \"b04d20be-e50b-432d-918e-db1854292fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:50.386524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.386488 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlp7s\" (UniqueName: \"kubernetes.io/projected/b04d20be-e50b-432d-918e-db1854292fc2-kube-api-access-dlp7s\") pod \"maas-api-revoke-sub-ab459dfd-bvdgh\" (UID: \"b04d20be-e50b-432d-918e-db1854292fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:50.483238 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.483152 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:50.793217 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.793140 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh"] Oct 31 17:38:50.796739 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:38:50.796709 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04d20be_e50b_432d_918e_db1854292fc2.slice/crio-4c3f6d6bcd8a495afb6192bfcfbb1e7a62df61a3498fa787ad4a5cf527021177 WatchSource:0}: Error finding container 4c3f6d6bcd8a495afb6192bfcfbb1e7a62df61a3498fa787ad4a5cf527021177: Status 404 returned error can't find the container with id 4c3f6d6bcd8a495afb6192bfcfbb1e7a62df61a3498fa787ad4a5cf527021177 Oct 31 17:38:50.947028 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:50.946992 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" event={"ID":"b04d20be-e50b-432d-918e-db1854292fc2","Type":"ContainerStarted","Data":"4c3f6d6bcd8a495afb6192bfcfbb1e7a62df61a3498fa787ad4a5cf527021177"} Oct 31 17:38:51.750523 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.750493 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9"] Oct 31 17:38:51.755114 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.755096 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:51.772979 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.772712 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9"] Oct 31 17:38:51.780923 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.780893 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-970bdea7-hcqp9\" (UID: \"7e6fc95a-6be4-45da-9ec7-3ae723278112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:51.781076 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.780933 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-970bdea7-hcqp9\" (UID: \"7e6fc95a-6be4-45da-9ec7-3ae723278112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:51.781180 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.781160 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lhzr\" (UniqueName: \"kubernetes.io/projected/7e6fc95a-6be4-45da-9ec7-3ae723278112-kube-api-access-5lhzr\") pod \"maas-api-revoke-sub-970bdea7-hcqp9\" (UID: \"7e6fc95a-6be4-45da-9ec7-3ae723278112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:51.882021 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.881988 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lhzr\" (UniqueName: \"kubernetes.io/projected/7e6fc95a-6be4-45da-9ec7-3ae723278112-kube-api-access-5lhzr\") pod \"maas-api-revoke-sub-970bdea7-hcqp9\" (UID: \"7e6fc95a-6be4-45da-9ec7-3ae723278112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:51.882214 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.882047 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-970bdea7-hcqp9\" (UID: \"7e6fc95a-6be4-45da-9ec7-3ae723278112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:51.882214 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.882086 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-970bdea7-hcqp9\" (UID: \"7e6fc95a-6be4-45da-9ec7-3ae723278112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:51.882934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.882902 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-970bdea7-hcqp9\" (UID: \"7e6fc95a-6be4-45da-9ec7-3ae723278112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:51.884631 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.884605 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-970bdea7-hcqp9\" (UID: \"7e6fc95a-6be4-45da-9ec7-3ae723278112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:51.899318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.899284 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lhzr\" (UniqueName: \"kubernetes.io/projected/7e6fc95a-6be4-45da-9ec7-3ae723278112-kube-api-access-5lhzr\") pod \"maas-api-revoke-sub-970bdea7-hcqp9\" (UID: \"7e6fc95a-6be4-45da-9ec7-3ae723278112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:51.955482 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.955442 7618 generic.go:356] "Generic (PLEG): container finished" podID="b04d20be-e50b-432d-918e-db1854292fc2" containerID="ce7bb5190a151d9a0583c390122789220e622e51b9a98c19c58ba1904b874d1d" exitCode=0 Oct 31 17:38:51.955635 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:51.955487 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" event={"ID":"b04d20be-e50b-432d-918e-db1854292fc2","Type":"ContainerDied","Data":"ce7bb5190a151d9a0583c390122789220e622e51b9a98c19c58ba1904b874d1d"} Oct 31 17:38:52.118331 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:52.118231 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:52.422053 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:52.422019 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9"] Oct 31 17:38:52.429415 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:38:52.429380 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6fc95a_6be4_45da_9ec7_3ae723278112.slice/crio-5e0ec3e1e8ec37c4feb66a67726dc79c8ee5bb067616d88062333fcdadde48c8 WatchSource:0}: Error finding container 5e0ec3e1e8ec37c4feb66a67726dc79c8ee5bb067616d88062333fcdadde48c8: Status 404 returned error can't find the container with id 5e0ec3e1e8ec37c4feb66a67726dc79c8ee5bb067616d88062333fcdadde48c8 Oct 31 17:38:52.963727 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:52.963686 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" event={"ID":"7e6fc95a-6be4-45da-9ec7-3ae723278112","Type":"ContainerStarted","Data":"de025b3b1f3c583cc5e1c5ce896a64e38705c2eeeda3eb158646bcab3c9366cd"} Oct 31 17:38:52.964225 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:52.963744 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" event={"ID":"7e6fc95a-6be4-45da-9ec7-3ae723278112","Type":"ContainerStarted","Data":"5e0ec3e1e8ec37c4feb66a67726dc79c8ee5bb067616d88062333fcdadde48c8"} Oct 31 17:38:52.997977 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:52.997854 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" podStartSLOduration=1.9978351559999998 podStartE2EDuration="1.997835156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:38:52.996858305 +0000 UTC m=+654.744832772" watchObservedRunningTime="2026-10-31 17:38:52.997835156 +0000 UTC m=+654.745809623" Oct 31 17:38:53.101981 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.101954 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:53.199055 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.199016 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-openshift-ca\") pod \"b04d20be-e50b-432d-918e-db1854292fc2\" (UID: \"b04d20be-e50b-432d-918e-db1854292fc2\") " Oct 31 17:38:53.199228 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.199087 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-xks-ca\") pod \"b04d20be-e50b-432d-918e-db1854292fc2\" (UID: \"b04d20be-e50b-432d-918e-db1854292fc2\") " Oct 31 17:38:53.199228 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.199119 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04d20be-e50b-432d-918e-db1854292fc2-kube-api-access-dlp7s\" (UniqueName: \"kubernetes.io/projected/b04d20be-e50b-432d-918e-db1854292fc2-kube-api-access-dlp7s\") pod \"b04d20be-e50b-432d-918e-db1854292fc2\" (UID: \"b04d20be-e50b-432d-918e-db1854292fc2\") " Oct 31 17:38:53.199513 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.199483 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-openshift-ca" pod "b04d20be-e50b-432d-918e-db1854292fc2" (UID: "b04d20be-e50b-432d-918e-db1854292fc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:38:53.199625 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.199602 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:53.201215 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.201190 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-xks-ca" pod "b04d20be-e50b-432d-918e-db1854292fc2" (UID: "b04d20be-e50b-432d-918e-db1854292fc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:38:53.201318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.201216 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04d20be-e50b-432d-918e-db1854292fc2-kube-api-access-dlp7s" pod "b04d20be-e50b-432d-918e-db1854292fc2" (UID: "b04d20be-e50b-432d-918e-db1854292fc2"). InnerVolumeSpecName "kube-api-access-dlp7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:38:53.300737 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.300696 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04d20be-e50b-432d-918e-db1854292fc2-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:53.300737 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.300731 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlp7s\" (UniqueName: \"kubernetes.io/projected/b04d20be-e50b-432d-918e-db1854292fc2-kube-api-access-dlp7s\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:53.970972 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.970936 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" Oct 31 17:38:53.971415 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.970936 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh" event={"ID":"b04d20be-e50b-432d-918e-db1854292fc2","Type":"ContainerDied","Data":"4c3f6d6bcd8a495afb6192bfcfbb1e7a62df61a3498fa787ad4a5cf527021177"} Oct 31 17:38:53.971415 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.971066 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c3f6d6bcd8a495afb6192bfcfbb1e7a62df61a3498fa787ad4a5cf527021177" Oct 31 17:38:53.972593 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.972569 7618 generic.go:356] "Generic (PLEG): container finished" podID="7e6fc95a-6be4-45da-9ec7-3ae723278112" containerID="de025b3b1f3c583cc5e1c5ce896a64e38705c2eeeda3eb158646bcab3c9366cd" exitCode=0 Oct 31 17:38:53.972714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:53.972632 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" event={"ID":"7e6fc95a-6be4-45da-9ec7-3ae723278112","Type":"ContainerDied","Data":"de025b3b1f3c583cc5e1c5ce896a64e38705c2eeeda3eb158646bcab3c9366cd"} Oct 31 17:38:55.131419 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.131385 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:55.221701 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.221660 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-xks-ca\") pod \"7e6fc95a-6be4-45da-9ec7-3ae723278112\" (UID: \"7e6fc95a-6be4-45da-9ec7-3ae723278112\") " Oct 31 17:38:55.221890 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.221841 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6fc95a-6be4-45da-9ec7-3ae723278112-kube-api-access-5lhzr\" (UniqueName: \"kubernetes.io/projected/7e6fc95a-6be4-45da-9ec7-3ae723278112-kube-api-access-5lhzr\") pod \"7e6fc95a-6be4-45da-9ec7-3ae723278112\" (UID: \"7e6fc95a-6be4-45da-9ec7-3ae723278112\") " Oct 31 17:38:55.221956 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.221908 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-openshift-ca\") pod \"7e6fc95a-6be4-45da-9ec7-3ae723278112\" (UID: \"7e6fc95a-6be4-45da-9ec7-3ae723278112\") " Oct 31 17:38:55.222443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.222417 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-openshift-ca" pod "7e6fc95a-6be4-45da-9ec7-3ae723278112" (UID: "7e6fc95a-6be4-45da-9ec7-3ae723278112"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:38:55.224212 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.224183 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-xks-ca" pod "7e6fc95a-6be4-45da-9ec7-3ae723278112" (UID: "7e6fc95a-6be4-45da-9ec7-3ae723278112"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:38:55.224457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.224431 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6fc95a-6be4-45da-9ec7-3ae723278112-kube-api-access-5lhzr" pod "7e6fc95a-6be4-45da-9ec7-3ae723278112" (UID: "7e6fc95a-6be4-45da-9ec7-3ae723278112"). InnerVolumeSpecName "kube-api-access-5lhzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:38:55.323678 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.323634 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lhzr\" (UniqueName: \"kubernetes.io/projected/7e6fc95a-6be4-45da-9ec7-3ae723278112-kube-api-access-5lhzr\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:55.323678 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.323665 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:55.323678 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.323676 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e6fc95a-6be4-45da-9ec7-3ae723278112-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:55.566295 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.566193 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd"] Oct 31 17:38:55.566708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.566692 7618 cpu_manager.go:414] "containerMap: removing container" podUID="b04d20be-e50b-432d-918e-db1854292fc2" containerName="revoke-keys" Oct 31 17:38:55.566783 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.566709 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04d20be-e50b-432d-918e-db1854292fc2" containerName="revoke-keys" Oct 31 17:38:55.566783 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.566727 7618 cpu_manager.go:414] "containerMap: removing container" podUID="7e6fc95a-6be4-45da-9ec7-3ae723278112" containerName="revoke-keys" Oct 31 17:38:55.566783 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.566733 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6fc95a-6be4-45da-9ec7-3ae723278112" containerName="revoke-keys" Oct 31 17:38:55.566945 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.566805 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04d20be-e50b-432d-918e-db1854292fc2" containerName="revoke-keys" Oct 31 17:38:55.566945 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.566817 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6fc95a-6be4-45da-9ec7-3ae723278112" containerName="revoke-keys" Oct 31 17:38:55.570347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.570321 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:38:55.587878 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.587849 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd"] Oct 31 17:38:55.627074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.627038 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10ac4fb-mpghd\" (UID: \"c532ee89-8206-40d2-ab8a-94a259e85911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:38:55.627074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.627073 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2jgr\" (UniqueName: \"kubernetes.io/projected/c532ee89-8206-40d2-ab8a-94a259e85911-kube-api-access-k2jgr\") pod \"maas-api-revoke-sub-e10ac4fb-mpghd\" (UID: \"c532ee89-8206-40d2-ab8a-94a259e85911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:38:55.627316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.627293 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10ac4fb-mpghd\" (UID: \"c532ee89-8206-40d2-ab8a-94a259e85911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:38:55.728947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.728909 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10ac4fb-mpghd\" (UID: \"c532ee89-8206-40d2-ab8a-94a259e85911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:38:55.728947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.728948 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jgr\" (UniqueName: \"kubernetes.io/projected/c532ee89-8206-40d2-ab8a-94a259e85911-kube-api-access-k2jgr\") pod \"maas-api-revoke-sub-e10ac4fb-mpghd\" (UID: \"c532ee89-8206-40d2-ab8a-94a259e85911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:38:55.729150 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.729113 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10ac4fb-mpghd\" (UID: \"c532ee89-8206-40d2-ab8a-94a259e85911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:38:55.729735 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.729710 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10ac4fb-mpghd\" (UID: \"c532ee89-8206-40d2-ab8a-94a259e85911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:38:55.731516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.731488 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10ac4fb-mpghd\" (UID: \"c532ee89-8206-40d2-ab8a-94a259e85911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:38:55.745584 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.745556 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jgr\" (UniqueName: \"kubernetes.io/projected/c532ee89-8206-40d2-ab8a-94a259e85911-kube-api-access-k2jgr\") pod \"maas-api-revoke-sub-e10ac4fb-mpghd\" (UID: \"c532ee89-8206-40d2-ab8a-94a259e85911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:38:55.884553 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.884471 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:38:55.986445 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.986411 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" event={"ID":"7e6fc95a-6be4-45da-9ec7-3ae723278112","Type":"ContainerDied","Data":"5e0ec3e1e8ec37c4feb66a67726dc79c8ee5bb067616d88062333fcdadde48c8"} Oct 31 17:38:55.986445 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.986432 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9" Oct 31 17:38:55.986640 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:55.986469 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e0ec3e1e8ec37c4feb66a67726dc79c8ee5bb067616d88062333fcdadde48c8" Oct 31 17:38:56.181247 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:56.181213 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd"] Oct 31 17:38:56.184053 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:38:56.184024 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc532ee89_8206_40d2_ab8a_94a259e85911.slice/crio-6f65869047b3af95185d24d739e6750af7c956d11e76b2e1deea624271b6cd2e WatchSource:0}: Error finding container 6f65869047b3af95185d24d739e6750af7c956d11e76b2e1deea624271b6cd2e: Status 404 returned error can't find the container with id 6f65869047b3af95185d24d739e6750af7c956d11e76b2e1deea624271b6cd2e Oct 31 17:38:56.993436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:56.993391 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" event={"ID":"c532ee89-8206-40d2-ab8a-94a259e85911","Type":"ContainerStarted","Data":"6d6a30430f8c22030c7131d11d74e4ed79f604110dd58cbf294c9dd503ea7580"} Oct 31 17:38:56.993648 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:56.993500 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" event={"ID":"c532ee89-8206-40d2-ab8a-94a259e85911","Type":"ContainerStarted","Data":"6f65869047b3af95185d24d739e6750af7c956d11e76b2e1deea624271b6cd2e"} Oct 31 17:38:57.026651 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:57.026599 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" podStartSLOduration=2.026585424 podStartE2EDuration="2.026585424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:38:57.026493286 +0000 UTC m=+658.774467752" watchObservedRunningTime="2026-10-31 17:38:57.026585424 +0000 UTC m=+658.774559891" Oct 31 17:38:57.935567 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:57.935534 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn"] Oct 31 17:38:57.939477 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:57.939460 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:38:57.960216 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:57.960184 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn"] Oct 31 17:38:58.000814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.000767 7618 generic.go:356] "Generic (PLEG): container finished" podID="c532ee89-8206-40d2-ab8a-94a259e85911" containerID="6d6a30430f8c22030c7131d11d74e4ed79f604110dd58cbf294c9dd503ea7580" exitCode=0 Oct 31 17:38:58.000982 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.000834 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" event={"ID":"c532ee89-8206-40d2-ab8a-94a259e85911","Type":"ContainerDied","Data":"6d6a30430f8c22030c7131d11d74e4ed79f604110dd58cbf294c9dd503ea7580"} Oct 31 17:38:58.056927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.056893 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b648f55-qtccn\" (UID: \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:38:58.056927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.056939 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkm79\" (UniqueName: \"kubernetes.io/projected/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-kube-api-access-zkm79\") pod \"maas-api-revoke-sub-4b648f55-qtccn\" (UID: \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:38:58.057147 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.056961 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b648f55-qtccn\" (UID: \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:38:58.157859 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.157817 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b648f55-qtccn\" (UID: \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:38:58.158044 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.157879 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkm79\" (UniqueName: \"kubernetes.io/projected/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-kube-api-access-zkm79\") pod \"maas-api-revoke-sub-4b648f55-qtccn\" (UID: \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:38:58.158044 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.157920 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b648f55-qtccn\" (UID: \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:38:58.158680 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.158652 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b648f55-qtccn\" (UID: \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:38:58.160309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.160278 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b648f55-qtccn\" (UID: \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:38:58.175284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.175222 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkm79\" (UniqueName: \"kubernetes.io/projected/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-kube-api-access-zkm79\") pod \"maas-api-revoke-sub-4b648f55-qtccn\" (UID: \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:38:58.254568 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.254474 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:38:58.550209 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:58.550120 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn"] Oct 31 17:38:58.553649 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:38:58.553615 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0cb58d2_8678_4a3c_860a_ccedb7f72f18.slice/crio-b20761ee847ad24e58ead4205055fa67b93bb2ff0589fe815b8844ecceec9754 WatchSource:0}: Error finding container b20761ee847ad24e58ead4205055fa67b93bb2ff0589fe815b8844ecceec9754: Status 404 returned error can't find the container with id b20761ee847ad24e58ead4205055fa67b93bb2ff0589fe815b8844ecceec9754 Oct 31 17:38:59.007668 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.007628 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" event={"ID":"d0cb58d2-8678-4a3c-860a-ccedb7f72f18","Type":"ContainerStarted","Data":"4b1ab4a0f3755252406caa6c229f7df0ab747a4adbbb7823c9c566c01ab2bab1"} Oct 31 17:38:59.008121 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.007701 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" event={"ID":"d0cb58d2-8678-4a3c-860a-ccedb7f72f18","Type":"ContainerStarted","Data":"b20761ee847ad24e58ead4205055fa67b93bb2ff0589fe815b8844ecceec9754"} Oct 31 17:38:59.040815 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.040748 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" podStartSLOduration=2.040729468 podStartE2EDuration="2.040729468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:38:59.040279367 +0000 UTC m=+660.788253834" watchObservedRunningTime="2026-10-31 17:38:59.040729468 +0000 UTC m=+660.788703936" Oct 31 17:38:59.151364 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.151337 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:38:59.270633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.270599 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-openshift-ca\") pod \"c532ee89-8206-40d2-ab8a-94a259e85911\" (UID: \"c532ee89-8206-40d2-ab8a-94a259e85911\") " Oct 31 17:38:59.270830 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.270770 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-xks-ca\") pod \"c532ee89-8206-40d2-ab8a-94a259e85911\" (UID: \"c532ee89-8206-40d2-ab8a-94a259e85911\") " Oct 31 17:38:59.270830 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.270810 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c532ee89-8206-40d2-ab8a-94a259e85911-kube-api-access-k2jgr\" (UniqueName: \"kubernetes.io/projected/c532ee89-8206-40d2-ab8a-94a259e85911-kube-api-access-k2jgr\") pod \"c532ee89-8206-40d2-ab8a-94a259e85911\" (UID: \"c532ee89-8206-40d2-ab8a-94a259e85911\") " Oct 31 17:38:59.271003 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.270971 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-openshift-ca" pod "c532ee89-8206-40d2-ab8a-94a259e85911" (UID: "c532ee89-8206-40d2-ab8a-94a259e85911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:38:59.271205 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.271185 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:59.272945 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.272917 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-xks-ca" pod "c532ee89-8206-40d2-ab8a-94a259e85911" (UID: "c532ee89-8206-40d2-ab8a-94a259e85911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:38:59.273178 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.273154 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c532ee89-8206-40d2-ab8a-94a259e85911-kube-api-access-k2jgr" pod "c532ee89-8206-40d2-ab8a-94a259e85911" (UID: "c532ee89-8206-40d2-ab8a-94a259e85911"). InnerVolumeSpecName "kube-api-access-k2jgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:38:59.372275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.372215 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c532ee89-8206-40d2-ab8a-94a259e85911-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:38:59.372275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:38:59.372247 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2jgr\" (UniqueName: \"kubernetes.io/projected/c532ee89-8206-40d2-ab8a-94a259e85911-kube-api-access-k2jgr\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:00.015140 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:00.015111 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" Oct 31 17:39:00.015628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:00.015145 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd" event={"ID":"c532ee89-8206-40d2-ab8a-94a259e85911","Type":"ContainerDied","Data":"6f65869047b3af95185d24d739e6750af7c956d11e76b2e1deea624271b6cd2e"} Oct 31 17:39:00.015628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:00.015201 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f65869047b3af95185d24d739e6750af7c956d11e76b2e1deea624271b6cd2e" Oct 31 17:39:00.017158 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:00.017129 7618 generic.go:356] "Generic (PLEG): container finished" podID="d0cb58d2-8678-4a3c-860a-ccedb7f72f18" containerID="4b1ab4a0f3755252406caa6c229f7df0ab747a4adbbb7823c9c566c01ab2bab1" exitCode=0 Oct 31 17:39:00.017281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:00.017190 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" event={"ID":"d0cb58d2-8678-4a3c-860a-ccedb7f72f18","Type":"ContainerDied","Data":"4b1ab4a0f3755252406caa6c229f7df0ab747a4adbbb7823c9c566c01ab2bab1"} Oct 31 17:39:01.169038 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.169012 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:39:01.293396 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.293303 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-openshift-ca\") pod \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\" (UID: \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\") " Oct 31 17:39:01.293546 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.293449 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-kube-api-access-zkm79\" (UniqueName: \"kubernetes.io/projected/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-kube-api-access-zkm79\") pod \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\" (UID: \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\") " Oct 31 17:39:01.293546 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.293472 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-xks-ca\") pod \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\" (UID: \"d0cb58d2-8678-4a3c-860a-ccedb7f72f18\") " Oct 31 17:39:01.293693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.293668 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-openshift-ca" pod "d0cb58d2-8678-4a3c-860a-ccedb7f72f18" (UID: "d0cb58d2-8678-4a3c-860a-ccedb7f72f18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:39:01.293876 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.293857 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:01.295617 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.295587 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-xks-ca" pod "d0cb58d2-8678-4a3c-860a-ccedb7f72f18" (UID: "d0cb58d2-8678-4a3c-860a-ccedb7f72f18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:39:01.295725 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.295702 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-kube-api-access-zkm79" pod "d0cb58d2-8678-4a3c-860a-ccedb7f72f18" (UID: "d0cb58d2-8678-4a3c-860a-ccedb7f72f18"). InnerVolumeSpecName "kube-api-access-zkm79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:39:01.364017 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.363978 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb"] Oct 31 17:39:01.364563 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.364547 7618 cpu_manager.go:414] "containerMap: removing container" podUID="c532ee89-8206-40d2-ab8a-94a259e85911" containerName="revoke-keys" Oct 31 17:39:01.364617 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.364564 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c532ee89-8206-40d2-ab8a-94a259e85911" containerName="revoke-keys" Oct 31 17:39:01.364617 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.364602 7618 cpu_manager.go:414] "containerMap: removing container" podUID="d0cb58d2-8678-4a3c-860a-ccedb7f72f18" containerName="revoke-keys" Oct 31 17:39:01.364617 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.364607 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0cb58d2-8678-4a3c-860a-ccedb7f72f18" containerName="revoke-keys" Oct 31 17:39:01.364707 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.364683 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="c532ee89-8206-40d2-ab8a-94a259e85911" containerName="revoke-keys" Oct 31 17:39:01.364707 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.364697 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0cb58d2-8678-4a3c-860a-ccedb7f72f18" containerName="revoke-keys" Oct 31 17:39:01.368358 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.368331 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.387139 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.387111 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-1c0bfd-openshift-default-dockercfg-27xm6\"" Oct 31 17:39:01.387334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.387284 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb"] Oct 31 17:39:01.394751 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.394725 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkm79\" (UniqueName: \"kubernetes.io/projected/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-kube-api-access-zkm79\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:01.394751 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.394753 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cb58d2-8678-4a3c-860a-ccedb7f72f18-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:01.496138 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.496098 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.496369 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.496147 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-credential-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.496443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.496375 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f68a151-a3f2-4609-beab-4cbe07577b59-istiod-ca-cert\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.496443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.496415 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-envoy\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.496579 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.496550 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-certs\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.496639 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.496609 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvnxf\" (UniqueName: \"kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-kube-api-access-pvnxf\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.496735 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.496699 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-token\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.496856 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.496826 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-podinfo\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.496963 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.496941 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-data\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598200 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598091 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598200 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598149 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-credential-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598200 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598194 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f68a151-a3f2-4609-beab-4cbe07577b59-istiod-ca-cert\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598544 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598320 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-envoy\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598544 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598385 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-certs\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598544 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598432 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvnxf\" (UniqueName: \"kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-kube-api-access-pvnxf\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598544 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598471 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-token\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598751 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598545 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-podinfo\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598751 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598596 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598751 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598600 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-data\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598751 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598673 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-credential-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598980 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598784 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-certs\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.598980 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.598836 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-data\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.599080 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.599028 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f68a151-a3f2-4609-beab-4cbe07577b59-istiod-ca-cert\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.600637 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.600592 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-envoy\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.600983 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.600964 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-podinfo\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.615053 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.615023 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-token\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.615173 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.615161 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvnxf\" (UniqueName: \"kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-kube-api-access-pvnxf\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:01.682735 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:01.682697 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:02.001560 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.001519 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb"] Oct 31 17:39:02.005575 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:39:02.005546 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f68a151_a3f2_4609_beab_4cbe07577b59.slice/crio-a98b21fb8260c5320bd91832436e74926cd66fbbf0835ba9dac37e41ee0bde21 WatchSource:0}: Error finding container a98b21fb8260c5320bd91832436e74926cd66fbbf0835ba9dac37e41ee0bde21: Status 404 returned error can't find the container with id a98b21fb8260c5320bd91832436e74926cd66fbbf0835ba9dac37e41ee0bde21 Oct 31 17:39:02.007934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.007901 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:02.008014 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.007969 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:02.008014 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.007998 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:02.032812 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.032768 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" event={"ID":"1f68a151-a3f2-4609-beab-4cbe07577b59","Type":"ContainerStarted","Data":"a98b21fb8260c5320bd91832436e74926cd66fbbf0835ba9dac37e41ee0bde21"} Oct 31 17:39:02.034624 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.034602 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" Oct 31 17:39:02.034752 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.034616 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn" event={"ID":"d0cb58d2-8678-4a3c-860a-ccedb7f72f18","Type":"ContainerDied","Data":"b20761ee847ad24e58ead4205055fa67b93bb2ff0589fe815b8844ecceec9754"} Oct 31 17:39:02.034752 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.034667 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b20761ee847ad24e58ead4205055fa67b93bb2ff0589fe815b8844ecceec9754" Oct 31 17:39:02.169667 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.169378 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl"] Oct 31 17:39:02.175669 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.175639 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:02.189145 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.188472 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:39:02.189145 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.188622 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:39:02.201109 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.201075 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl"] Oct 31 17:39:02.307920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.307825 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-615c753c-fh6zl\" (UID: \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:02.307920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.307880 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s6w4\" (UniqueName: \"kubernetes.io/projected/3d5850b4-8cf8-47e6-baeb-2d40def410b9-kube-api-access-2s6w4\") pod \"maas-api-revoke-sub-615c753c-fh6zl\" (UID: \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:02.308124 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.308099 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-615c753c-fh6zl\" (UID: \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:02.408814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.408775 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-615c753c-fh6zl\" (UID: \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:02.409010 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.408880 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-615c753c-fh6zl\" (UID: \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:02.409010 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.408906 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s6w4\" (UniqueName: \"kubernetes.io/projected/3d5850b4-8cf8-47e6-baeb-2d40def410b9-kube-api-access-2s6w4\") pod \"maas-api-revoke-sub-615c753c-fh6zl\" (UID: \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:02.409719 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.409635 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-615c753c-fh6zl\" (UID: \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:02.411208 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.411184 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-615c753c-fh6zl\" (UID: \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:02.431824 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.431740 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s6w4\" (UniqueName: \"kubernetes.io/projected/3d5850b4-8cf8-47e6-baeb-2d40def410b9-kube-api-access-2s6w4\") pod \"maas-api-revoke-sub-615c753c-fh6zl\" (UID: \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:02.507179 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.507141 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:02.829496 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:02.827280 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl"] Oct 31 17:39:02.829496 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:39:02.828465 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5850b4_8cf8_47e6_baeb_2d40def410b9.slice/crio-c8d27250b48df8120c41ab6781acb0cb6866807522b7caef3da0432fdfcc65d2 WatchSource:0}: Error finding container c8d27250b48df8120c41ab6781acb0cb6866807522b7caef3da0432fdfcc65d2: Status 404 returned error can't find the container with id c8d27250b48df8120c41ab6781acb0cb6866807522b7caef3da0432fdfcc65d2 Oct 31 17:39:03.065813 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:03.065762 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" event={"ID":"3d5850b4-8cf8-47e6-baeb-2d40def410b9","Type":"ContainerStarted","Data":"c8d27250b48df8120c41ab6781acb0cb6866807522b7caef3da0432fdfcc65d2"} Oct 31 17:39:03.067940 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:03.067906 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" event={"ID":"1f68a151-a3f2-4609-beab-4cbe07577b59","Type":"ContainerStarted","Data":"62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f"} Oct 31 17:39:03.102099 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:03.102008 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:39:03.113026 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:03.112953 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" podStartSLOduration=2.11293416 podStartE2EDuration="2.11293416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:03.11004441 +0000 UTC m=+664.858018874" watchObservedRunningTime="2026-10-31 17:39:03.11293416 +0000 UTC m=+664.860908641" Oct 31 17:39:03.683514 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:03.683474 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:03.684839 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:03.684806 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 17:39:03.684965 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:03.684864 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" podUID="1f68a151-a3f2-4609-beab-4cbe07577b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 17:39:04.075337 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:04.075222 7618 generic.go:356] "Generic (PLEG): container finished" podID="3d5850b4-8cf8-47e6-baeb-2d40def410b9" containerID="f994158955406d91c1df1b49cd140b8a82e10ec1b45303cb21382ae121d2a765" exitCode=0 Oct 31 17:39:04.075337 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:04.075309 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" event={"ID":"3d5850b4-8cf8-47e6-baeb-2d40def410b9","Type":"ContainerDied","Data":"f994158955406d91c1df1b49cd140b8a82e10ec1b45303cb21382ae121d2a765"} Oct 31 17:39:04.684429 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:04.683950 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 17:39:04.684429 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:04.684013 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" podUID="1f68a151-a3f2-4609-beab-4cbe07577b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 17:39:05.367901 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.367820 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq"] Oct 31 17:39:05.392434 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.389831 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.407433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.406816 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0e59ca\"/\"openshift-service-ca.crt\"" Oct 31 17:39:05.412776 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.411670 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq"] Oct 31 17:39:05.425470 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.425144 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0e59ca\"/\"kube-root-ca.crt\"" Oct 31 17:39:05.425470 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.425333 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0e59ca\"/\"default-dockercfg-pwznp\"" Oct 31 17:39:05.459203 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.459157 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0e59ca\"/\"distinct-w1-0e59ca-kserve-self-signed-certs\"" Oct 31 17:39:05.469288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.461432 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:05.504725 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.504683 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-model-cache\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.505265 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.505225 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-home\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.505429 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.505414 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b962c13d-7616-4971-9ae1-5b36c812e296-tls-certs\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.505567 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.505553 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-tmp-dir\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.505694 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.505680 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-dshm\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.505882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.505865 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhsc7\" (UniqueName: \"kubernetes.io/projected/b962c13d-7616-4971-9ae1-5b36c812e296-kube-api-access-jhsc7\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.608590 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-openshift-ca\") pod \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\" (UID: \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\") " Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.608703 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5850b4-8cf8-47e6-baeb-2d40def410b9-kube-api-access-2s6w4\" (UniqueName: \"kubernetes.io/projected/3d5850b4-8cf8-47e6-baeb-2d40def410b9-kube-api-access-2s6w4\") pod \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\" (UID: \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\") " Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.608771 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-xks-ca\") pod \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\" (UID: \"3d5850b4-8cf8-47e6-baeb-2d40def410b9\") " Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.608993 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-model-cache\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.609047 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-home\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.609079 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b962c13d-7616-4971-9ae1-5b36c812e296-tls-certs\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.609110 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-tmp-dir\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.609149 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-dshm\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.609221 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhsc7\" (UniqueName: \"kubernetes.io/projected/b962c13d-7616-4971-9ae1-5b36c812e296-kube-api-access-jhsc7\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.610036 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-openshift-ca" pod "3d5850b4-8cf8-47e6-baeb-2d40def410b9" (UID: "3d5850b4-8cf8-47e6-baeb-2d40def410b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.610924 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-home\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.611613 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-model-cache\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.620328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.611941 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-tmp-dir\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.695275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.692358 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b962c13d-7616-4971-9ae1-5b36c812e296-tls-certs\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.695275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.692694 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-xks-ca" pod "3d5850b4-8cf8-47e6-baeb-2d40def410b9" (UID: "3d5850b4-8cf8-47e6-baeb-2d40def410b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:39:05.695275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.692784 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 17:39:05.695275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.692828 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" podUID="1f68a151-a3f2-4609-beab-4cbe07577b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 17:39:05.695921 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.695558 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-dshm\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.723273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.722626 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:05.723273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.722661 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d5850b4-8cf8-47e6-baeb-2d40def410b9-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:05.755062 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.755013 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhsc7\" (UniqueName: \"kubernetes.io/projected/b962c13d-7616-4971-9ae1-5b36c812e296-kube-api-access-jhsc7\") pod \"distinct-w1-0e59ca-kserve-8544d68ddf-9zntq\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:05.755829 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.755779 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5850b4-8cf8-47e6-baeb-2d40def410b9-kube-api-access-2s6w4" pod "3d5850b4-8cf8-47e6-baeb-2d40def410b9" (UID: "3d5850b4-8cf8-47e6-baeb-2d40def410b9"). InnerVolumeSpecName "kube-api-access-2s6w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:39:05.828409 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.828362 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s6w4\" (UniqueName: \"kubernetes.io/projected/3d5850b4-8cf8-47e6-baeb-2d40def410b9-kube-api-access-2s6w4\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:05.845871 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:05.845504 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:06.114268 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:06.110438 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" Oct 31 17:39:06.114268 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:06.110480 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl" event={"ID":"3d5850b4-8cf8-47e6-baeb-2d40def410b9","Type":"ContainerDied","Data":"c8d27250b48df8120c41ab6781acb0cb6866807522b7caef3da0432fdfcc65d2"} Oct 31 17:39:06.114268 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:06.110536 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8d27250b48df8120c41ab6781acb0cb6866807522b7caef3da0432fdfcc65d2" Oct 31 17:39:06.515109 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:06.510595 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq"] Oct 31 17:39:06.688276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:06.684988 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 17:39:06.688276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:06.685044 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" podUID="1f68a151-a3f2-4609-beab-4cbe07577b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 17:39:07.146999 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:07.143740 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" event={"ID":"b962c13d-7616-4971-9ae1-5b36c812e296","Type":"ContainerStarted","Data":"637737e0a7edd40a4b2d4c3433b35e8fb40e394073b38b09a14a5ce23a9b86e0"} Oct 31 17:39:07.154273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:07.153187 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kd6mw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:39:07.154273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:07.153265 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" podUID="df608734-a9f8-40c6-9430-1b25fa3389e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:07.691278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:07.689566 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 17:39:07.691278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:07.689633 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" podUID="1f68a151-a3f2-4609-beab-4cbe07577b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 17:39:08.154447 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.154401 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" event={"ID":"b962c13d-7616-4971-9ae1-5b36c812e296","Type":"ContainerStarted","Data":"8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4"} Oct 31 17:39:08.191157 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.190965 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podStartSLOduration=3.190951477 podStartE2EDuration="3.190951477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:08.190241566 +0000 UTC m=+669.938216046" watchObservedRunningTime="2026-10-31 17:39:08.190951477 +0000 UTC m=+669.938925943" Oct 31 17:39:08.573934 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:39:08.573849 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:39:08.684091 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.684049 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 17:39:08.684345 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.684124 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" podUID="1f68a151-a3f2-4609-beab-4cbe07577b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 17:39:08.816148 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.816105 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7"] Oct 31 17:39:08.816947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.816920 7618 cpu_manager.go:414] "containerMap: removing container" podUID="3d5850b4-8cf8-47e6-baeb-2d40def410b9" containerName="revoke-keys" Oct 31 17:39:08.817050 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.816960 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5850b4-8cf8-47e6-baeb-2d40def410b9" containerName="revoke-keys" Oct 31 17:39:08.817111 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.817098 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5850b4-8cf8-47e6-baeb-2d40def410b9" containerName="revoke-keys" Oct 31 17:39:08.821182 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.821157 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:08.836447 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.836416 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7"] Oct 31 17:39:08.836583 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.836558 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:39:08.842820 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.842771 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:39:08.846877 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.846846 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:08.919753 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.919705 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7wb\" (UniqueName: \"kubernetes.io/projected/09698b60-c579-4a2e-a3ef-124b69b21438-kube-api-access-qm7wb\") pod \"maas-api-revoke-sub-ef789c02-4tcd7\" (UID: \"09698b60-c579-4a2e-a3ef-124b69b21438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:08.919961 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.919764 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef789c02-4tcd7\" (UID: \"09698b60-c579-4a2e-a3ef-124b69b21438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:08.919961 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:08.919859 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef789c02-4tcd7\" (UID: \"09698b60-c579-4a2e-a3ef-124b69b21438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:09.020578 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:09.020544 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7wb\" (UniqueName: \"kubernetes.io/projected/09698b60-c579-4a2e-a3ef-124b69b21438-kube-api-access-qm7wb\") pod \"maas-api-revoke-sub-ef789c02-4tcd7\" (UID: \"09698b60-c579-4a2e-a3ef-124b69b21438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:09.020789 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:09.020599 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef789c02-4tcd7\" (UID: \"09698b60-c579-4a2e-a3ef-124b69b21438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:09.020789 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:09.020635 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef789c02-4tcd7\" (UID: \"09698b60-c579-4a2e-a3ef-124b69b21438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:09.021587 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:09.021549 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef789c02-4tcd7\" (UID: \"09698b60-c579-4a2e-a3ef-124b69b21438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:09.023267 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:09.023225 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef789c02-4tcd7\" (UID: \"09698b60-c579-4a2e-a3ef-124b69b21438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:09.038024 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:09.037989 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7wb\" (UniqueName: \"kubernetes.io/projected/09698b60-c579-4a2e-a3ef-124b69b21438-kube-api-access-qm7wb\") pod \"maas-api-revoke-sub-ef789c02-4tcd7\" (UID: \"09698b60-c579-4a2e-a3ef-124b69b21438\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:09.137857 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:09.137768 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:09.444220 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:09.444162 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7"] Oct 31 17:39:09.450054 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:39:09.450025 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09698b60_c579_4a2e_a3ef_124b69b21438.slice/crio-a9877af0da2e532a3d4c4fb7b25aef4c9f5be6a3e6ef5bc34c15149a044ed078 WatchSource:0}: Error finding container a9877af0da2e532a3d4c4fb7b25aef4c9f5be6a3e6ef5bc34c15149a044ed078: Status 404 returned error can't find the container with id a9877af0da2e532a3d4c4fb7b25aef4c9f5be6a3e6ef5bc34c15149a044ed078 Oct 31 17:39:09.689181 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:09.689147 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:09.690303 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:09.690054 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:09.691050 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:09.691025 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:39:10.171438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.171395 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" event={"ID":"09698b60-c579-4a2e-a3ef-124b69b21438","Type":"ContainerStarted","Data":"0bc13eea3db8e1a32b2630980a673b6755dc650131a1e1ab58a2c0e1ebe4ad10"} Oct 31 17:39:10.171618 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.171457 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" event={"ID":"09698b60-c579-4a2e-a3ef-124b69b21438","Type":"ContainerStarted","Data":"a9877af0da2e532a3d4c4fb7b25aef4c9f5be6a3e6ef5bc34c15149a044ed078"} Oct 31 17:39:10.205383 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.205329 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" podStartSLOduration=2.205313528 podStartE2EDuration="2.205313528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:10.204700485 +0000 UTC m=+671.952674959" watchObservedRunningTime="2026-10-31 17:39:10.205313528 +0000 UTC m=+671.953287995" Oct 31 17:39:10.623908 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.623875 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd"] Oct 31 17:39:10.628549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.628518 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:10.643794 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.643599 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd"] Oct 31 17:39:10.743281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.743187 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d87b62-wr2jd\" (UID: \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:10.743433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.743338 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d87b62-wr2jd\" (UID: \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:10.743433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.743390 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsp5x\" (UniqueName: \"kubernetes.io/projected/4a8c99be-1b4f-4031-8126-2a9a1abe3757-kube-api-access-hsp5x\") pod \"maas-api-revoke-sub-a3d87b62-wr2jd\" (UID: \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:10.844298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.844248 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d87b62-wr2jd\" (UID: \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:10.844505 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.844391 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d87b62-wr2jd\" (UID: \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:10.844505 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.844423 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsp5x\" (UniqueName: \"kubernetes.io/projected/4a8c99be-1b4f-4031-8126-2a9a1abe3757-kube-api-access-hsp5x\") pod \"maas-api-revoke-sub-a3d87b62-wr2jd\" (UID: \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:10.845094 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.845067 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d87b62-wr2jd\" (UID: \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:10.846904 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.846878 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d87b62-wr2jd\" (UID: \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:10.861201 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.861170 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsp5x\" (UniqueName: \"kubernetes.io/projected/4a8c99be-1b4f-4031-8126-2a9a1abe3757-kube-api-access-hsp5x\") pod \"maas-api-revoke-sub-a3d87b62-wr2jd\" (UID: \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:10.941996 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:10.941950 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:11.178800 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:11.178756 7618 generic.go:356] "Generic (PLEG): container finished" podID="09698b60-c579-4a2e-a3ef-124b69b21438" containerID="0bc13eea3db8e1a32b2630980a673b6755dc650131a1e1ab58a2c0e1ebe4ad10" exitCode=0 Oct 31 17:39:11.178958 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:11.178838 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" event={"ID":"09698b60-c579-4a2e-a3ef-124b69b21438","Type":"ContainerDied","Data":"0bc13eea3db8e1a32b2630980a673b6755dc650131a1e1ab58a2c0e1ebe4ad10"} Oct 31 17:39:11.251060 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:11.250698 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd"] Oct 31 17:39:12.185987 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.185950 7618 generic.go:356] "Generic (PLEG): container finished" podID="4a8c99be-1b4f-4031-8126-2a9a1abe3757" containerID="d76f9a0eb4234d7bd7d3fdd56e0056a3946eb75ba88fd765bfd34386bfd1a82c" exitCode=0 Oct 31 17:39:12.186570 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.186034 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" event={"ID":"4a8c99be-1b4f-4031-8126-2a9a1abe3757","Type":"ContainerDied","Data":"d76f9a0eb4234d7bd7d3fdd56e0056a3946eb75ba88fd765bfd34386bfd1a82c"} Oct 31 17:39:12.186570 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.186091 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" event={"ID":"4a8c99be-1b4f-4031-8126-2a9a1abe3757","Type":"ContainerStarted","Data":"ec125e0503872b9f4413c72d529cdeb6beb2978468ca91cb0c77f5e9378b84b8"} Oct 31 17:39:12.329960 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.329744 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:12.368124 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.364396 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09698b60-c579-4a2e-a3ef-124b69b21438-kube-api-access-qm7wb\" (UniqueName: \"kubernetes.io/projected/09698b60-c579-4a2e-a3ef-124b69b21438-kube-api-access-qm7wb\") pod \"09698b60-c579-4a2e-a3ef-124b69b21438\" (UID: \"09698b60-c579-4a2e-a3ef-124b69b21438\") " Oct 31 17:39:12.368124 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.367428 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-xks-ca\") pod \"09698b60-c579-4a2e-a3ef-124b69b21438\" (UID: \"09698b60-c579-4a2e-a3ef-124b69b21438\") " Oct 31 17:39:12.368124 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.367494 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-openshift-ca\") pod \"09698b60-c579-4a2e-a3ef-124b69b21438\" (UID: \"09698b60-c579-4a2e-a3ef-124b69b21438\") " Oct 31 17:39:12.368493 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.368322 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-openshift-ca" pod "09698b60-c579-4a2e-a3ef-124b69b21438" (UID: "09698b60-c579-4a2e-a3ef-124b69b21438"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:39:12.373667 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.373622 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-xks-ca" pod "09698b60-c579-4a2e-a3ef-124b69b21438" (UID: "09698b60-c579-4a2e-a3ef-124b69b21438"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:39:12.374105 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.373809 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09698b60-c579-4a2e-a3ef-124b69b21438-kube-api-access-qm7wb" pod "09698b60-c579-4a2e-a3ef-124b69b21438" (UID: "09698b60-c579-4a2e-a3ef-124b69b21438"). InnerVolumeSpecName "kube-api-access-qm7wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:39:12.468941 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.468896 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:12.468941 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.468928 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09698b60-c579-4a2e-a3ef-124b69b21438-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:12.468941 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:12.468944 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm7wb\" (UniqueName: \"kubernetes.io/projected/09698b60-c579-4a2e-a3ef-124b69b21438-kube-api-access-qm7wb\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:13.193797 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.193751 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" event={"ID":"09698b60-c579-4a2e-a3ef-124b69b21438","Type":"ContainerDied","Data":"a9877af0da2e532a3d4c4fb7b25aef4c9f5be6a3e6ef5bc34c15149a044ed078"} Oct 31 17:39:13.194212 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.193817 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9877af0da2e532a3d4c4fb7b25aef4c9f5be6a3e6ef5bc34c15149a044ed078" Oct 31 17:39:13.194212 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.193775 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7" Oct 31 17:39:13.339066 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.339036 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:13.379574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.379537 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-xks-ca\") pod \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\" (UID: \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\") " Oct 31 17:39:13.379835 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.379681 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-openshift-ca\") pod \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\" (UID: \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\") " Oct 31 17:39:13.379835 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.379765 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a8c99be-1b4f-4031-8126-2a9a1abe3757-kube-api-access-hsp5x\" (UniqueName: \"kubernetes.io/projected/4a8c99be-1b4f-4031-8126-2a9a1abe3757-kube-api-access-hsp5x\") pod \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\" (UID: \"4a8c99be-1b4f-4031-8126-2a9a1abe3757\") " Oct 31 17:39:13.380150 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.380118 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-openshift-ca" pod "4a8c99be-1b4f-4031-8126-2a9a1abe3757" (UID: "4a8c99be-1b4f-4031-8126-2a9a1abe3757"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:39:13.382645 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.382617 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-xks-ca" pod "4a8c99be-1b4f-4031-8126-2a9a1abe3757" (UID: "4a8c99be-1b4f-4031-8126-2a9a1abe3757"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:39:13.382758 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.382732 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8c99be-1b4f-4031-8126-2a9a1abe3757-kube-api-access-hsp5x" pod "4a8c99be-1b4f-4031-8126-2a9a1abe3757" (UID: "4a8c99be-1b4f-4031-8126-2a9a1abe3757"). InnerVolumeSpecName "kube-api-access-hsp5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:39:13.481418 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.481334 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:13.481418 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.481363 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsp5x\" (UniqueName: \"kubernetes.io/projected/4a8c99be-1b4f-4031-8126-2a9a1abe3757-kube-api-access-hsp5x\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:13.481418 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.481375 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8c99be-1b4f-4031-8126-2a9a1abe3757-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:13.999284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.999213 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt"] Oct 31 17:39:13.999817 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.999796 7618 cpu_manager.go:414] "containerMap: removing container" podUID="09698b60-c579-4a2e-a3ef-124b69b21438" containerName="revoke-keys" Oct 31 17:39:13.999919 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.999820 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09698b60-c579-4a2e-a3ef-124b69b21438" containerName="revoke-keys" Oct 31 17:39:13.999919 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.999849 7618 cpu_manager.go:414] "containerMap: removing container" podUID="4a8c99be-1b4f-4031-8126-2a9a1abe3757" containerName="revoke-keys" Oct 31 17:39:13.999919 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.999857 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a8c99be-1b4f-4031-8126-2a9a1abe3757" containerName="revoke-keys" Oct 31 17:39:14.000074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:13.999992 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="09698b60-c579-4a2e-a3ef-124b69b21438" containerName="revoke-keys" Oct 31 17:39:14.000074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.000007 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a8c99be-1b4f-4031-8126-2a9a1abe3757" containerName="revoke-keys" Oct 31 17:39:14.003156 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.003133 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:14.020419 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.020385 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt"] Oct 31 17:39:14.087646 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.087608 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1188609b-eccc-4717-8d36-e8252469621e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e4ade94-l4hqt\" (UID: \"1188609b-eccc-4717-8d36-e8252469621e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:14.087846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.087823 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snmck\" (UniqueName: \"kubernetes.io/projected/1188609b-eccc-4717-8d36-e8252469621e-kube-api-access-snmck\") pod \"maas-api-revoke-sub-0e4ade94-l4hqt\" (UID: \"1188609b-eccc-4717-8d36-e8252469621e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:14.087891 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.087859 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1188609b-eccc-4717-8d36-e8252469621e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e4ade94-l4hqt\" (UID: \"1188609b-eccc-4717-8d36-e8252469621e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:14.189411 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.189361 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snmck\" (UniqueName: \"kubernetes.io/projected/1188609b-eccc-4717-8d36-e8252469621e-kube-api-access-snmck\") pod \"maas-api-revoke-sub-0e4ade94-l4hqt\" (UID: \"1188609b-eccc-4717-8d36-e8252469621e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:14.189411 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.189412 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1188609b-eccc-4717-8d36-e8252469621e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e4ade94-l4hqt\" (UID: \"1188609b-eccc-4717-8d36-e8252469621e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:14.189676 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.189459 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1188609b-eccc-4717-8d36-e8252469621e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e4ade94-l4hqt\" (UID: \"1188609b-eccc-4717-8d36-e8252469621e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:14.190176 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.190151 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1188609b-eccc-4717-8d36-e8252469621e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e4ade94-l4hqt\" (UID: \"1188609b-eccc-4717-8d36-e8252469621e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:14.191754 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.191729 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1188609b-eccc-4717-8d36-e8252469621e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e4ade94-l4hqt\" (UID: \"1188609b-eccc-4717-8d36-e8252469621e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:14.211433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.211387 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" event={"ID":"4a8c99be-1b4f-4031-8126-2a9a1abe3757","Type":"ContainerDied","Data":"ec125e0503872b9f4413c72d529cdeb6beb2978468ca91cb0c77f5e9378b84b8"} Oct 31 17:39:14.211894 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.211498 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec125e0503872b9f4413c72d529cdeb6beb2978468ca91cb0c77f5e9378b84b8" Oct 31 17:39:14.211894 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.211661 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd" Oct 31 17:39:14.215095 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.215068 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snmck\" (UniqueName: \"kubernetes.io/projected/1188609b-eccc-4717-8d36-e8252469621e-kube-api-access-snmck\") pod \"maas-api-revoke-sub-0e4ade94-l4hqt\" (UID: \"1188609b-eccc-4717-8d36-e8252469621e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:14.316453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.316375 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:14.624040 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:14.623939 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt"] Oct 31 17:39:14.627239 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:39:14.626532 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1188609b_eccc_4717_8d36_e8252469621e.slice/crio-39246af7a2c93cd65ca56c708bad86c9e125cc1b0995ef58e392df6fd99e8310 WatchSource:0}: Error finding container 39246af7a2c93cd65ca56c708bad86c9e125cc1b0995ef58e392df6fd99e8310: Status 404 returned error can't find the container with id 39246af7a2c93cd65ca56c708bad86c9e125cc1b0995ef58e392df6fd99e8310 Oct 31 17:39:15.218552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:15.218460 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" event={"ID":"1188609b-eccc-4717-8d36-e8252469621e","Type":"ContainerStarted","Data":"d612dd3653cd4e4361ef87de5e52c497c38fe25df6dbb11f822195086e629d92"} Oct 31 17:39:15.218552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:15.218510 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" event={"ID":"1188609b-eccc-4717-8d36-e8252469621e","Type":"ContainerStarted","Data":"39246af7a2c93cd65ca56c708bad86c9e125cc1b0995ef58e392df6fd99e8310"} Oct 31 17:39:15.255745 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:15.255690 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" podStartSLOduration=2.255672216 podStartE2EDuration="2.255672216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:15.254248504 +0000 UTC m=+677.002222997" watchObservedRunningTime="2026-10-31 17:39:15.255672216 +0000 UTC m=+677.003646687" Oct 31 17:39:15.847400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:15.846547 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:15.861954 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:15.861921 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:16.232665 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.232626 7618 generic.go:356] "Generic (PLEG): container finished" podID="1188609b-eccc-4717-8d36-e8252469621e" containerID="d612dd3653cd4e4361ef87de5e52c497c38fe25df6dbb11f822195086e629d92" exitCode=0 Oct 31 17:39:16.233184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.232720 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" event={"ID":"1188609b-eccc-4717-8d36-e8252469621e","Type":"ContainerDied","Data":"d612dd3653cd4e4361ef87de5e52c497c38fe25df6dbb11f822195086e629d92"} Oct 31 17:39:16.250217 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.250185 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:16.799277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.797374 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b"] Oct 31 17:39:16.802590 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.802563 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:16.819720 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.817298 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ee816e-openshift-default-dockercfg-mnsfm\"" Oct 31 17:39:16.835274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.830986 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b"] Oct 31 17:39:16.933094 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.932760 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-data\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:16.933094 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.932925 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:16.933094 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.933025 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-certs\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:16.933094 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.933091 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446910ca-78af-4c6f-8196-56c209b33636-istio-podinfo\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:16.933569 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.933125 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446910ca-78af-4c6f-8196-56c209b33636-istiod-ca-cert\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:16.933569 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.933166 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-envoy\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:16.933569 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.933202 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-credential-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:16.933569 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.933285 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2xjj\" (UniqueName: \"kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-kube-api-access-q2xjj\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:16.933569 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:16.933313 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-istio-token\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.034271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.034221 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-credential-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.034481 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.034311 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2xjj\" (UniqueName: \"kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-kube-api-access-q2xjj\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.034481 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.034340 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-istio-token\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.034481 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.034379 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-data\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.034481 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.034451 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.034719 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.034533 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-certs\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.034719 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.034567 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446910ca-78af-4c6f-8196-56c209b33636-istio-podinfo\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.034719 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.034601 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446910ca-78af-4c6f-8196-56c209b33636-istiod-ca-cert\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.034719 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.034644 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-envoy\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.034946 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.034745 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-credential-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.035409 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.035039 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.035409 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.035351 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-certs\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.035780 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.035751 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446910ca-78af-4c6f-8196-56c209b33636-istiod-ca-cert\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.035924 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.035761 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-data\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.037095 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.037070 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-envoy\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.037521 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.037503 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446910ca-78af-4c6f-8196-56c209b33636-istio-podinfo\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.051512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.051442 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-istio-token\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.051750 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.051724 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2xjj\" (UniqueName: \"kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-kube-api-access-q2xjj\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.120553 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.120510 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:17.481381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.480177 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b"] Oct 31 17:39:17.486504 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.486450 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:17.486597 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.486578 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:17.486661 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.486643 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:17.500538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.500510 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:17.656310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.655299 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1188609b-eccc-4717-8d36-e8252469621e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1188609b-eccc-4717-8d36-e8252469621e-maas-api-xks-ca\") pod \"1188609b-eccc-4717-8d36-e8252469621e\" (UID: \"1188609b-eccc-4717-8d36-e8252469621e\") " Oct 31 17:39:17.656310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.655426 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1188609b-eccc-4717-8d36-e8252469621e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1188609b-eccc-4717-8d36-e8252469621e-maas-api-openshift-ca\") pod \"1188609b-eccc-4717-8d36-e8252469621e\" (UID: \"1188609b-eccc-4717-8d36-e8252469621e\") " Oct 31 17:39:17.656310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.655489 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1188609b-eccc-4717-8d36-e8252469621e-kube-api-access-snmck\" (UniqueName: \"kubernetes.io/projected/1188609b-eccc-4717-8d36-e8252469621e-kube-api-access-snmck\") pod \"1188609b-eccc-4717-8d36-e8252469621e\" (UID: \"1188609b-eccc-4717-8d36-e8252469621e\") " Oct 31 17:39:17.657371 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.657325 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1188609b-eccc-4717-8d36-e8252469621e-maas-api-openshift-ca" pod "1188609b-eccc-4717-8d36-e8252469621e" (UID: "1188609b-eccc-4717-8d36-e8252469621e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:39:17.671307 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.671238 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1188609b-eccc-4717-8d36-e8252469621e-maas-api-xks-ca" pod "1188609b-eccc-4717-8d36-e8252469621e" (UID: "1188609b-eccc-4717-8d36-e8252469621e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:39:17.675453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.675414 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1188609b-eccc-4717-8d36-e8252469621e-kube-api-access-snmck" pod "1188609b-eccc-4717-8d36-e8252469621e" (UID: "1188609b-eccc-4717-8d36-e8252469621e"). InnerVolumeSpecName "kube-api-access-snmck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:39:17.757277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.756837 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1188609b-eccc-4717-8d36-e8252469621e-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:17.757277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.756869 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1188609b-eccc-4717-8d36-e8252469621e-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:17.757277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:17.756882 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snmck\" (UniqueName: \"kubernetes.io/projected/1188609b-eccc-4717-8d36-e8252469621e-kube-api-access-snmck\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:18.293587 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:18.293552 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" event={"ID":"1188609b-eccc-4717-8d36-e8252469621e","Type":"ContainerDied","Data":"39246af7a2c93cd65ca56c708bad86c9e125cc1b0995ef58e392df6fd99e8310"} Oct 31 17:39:18.293792 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:18.293644 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39246af7a2c93cd65ca56c708bad86c9e125cc1b0995ef58e392df6fd99e8310" Oct 31 17:39:18.293792 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:18.293720 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt" Oct 31 17:39:18.299322 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:18.299285 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" event={"ID":"446910ca-78af-4c6f-8196-56c209b33636","Type":"ContainerStarted","Data":"544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0"} Oct 31 17:39:18.299499 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:18.299342 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" event={"ID":"446910ca-78af-4c6f-8196-56c209b33636","Type":"ContainerStarted","Data":"75b4e829d1b339ff01c739f8de9e09086984906a062f92e6f7e2f86a3ab18c4d"} Oct 31 17:39:19.132277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:19.127320 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:19.132277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:19.127520 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 17:39:19.132277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:19.127568 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" podUID="446910ca-78af-4c6f-8196-56c209b33636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 17:39:20.129278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.122678 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 17:39:20.129278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.122745 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" podUID="446910ca-78af-4c6f-8196-56c209b33636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 17:39:20.673710 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.673656 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" podStartSLOduration=4.673640719 podStartE2EDuration="4.673640719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:18.341605916 +0000 UTC m=+680.089580384" watchObservedRunningTime="2026-10-31 17:39:20.673640719 +0000 UTC m=+682.421615187" Oct 31 17:39:20.675143 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.675115 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7"] Oct 31 17:39:20.675648 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.675634 7618 cpu_manager.go:414] "containerMap: removing container" podUID="1188609b-eccc-4717-8d36-e8252469621e" containerName="revoke-keys" Oct 31 17:39:20.675697 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.675650 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1188609b-eccc-4717-8d36-e8252469621e" containerName="revoke-keys" Oct 31 17:39:20.675770 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.675761 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="1188609b-eccc-4717-8d36-e8252469621e" containerName="revoke-keys" Oct 31 17:39:20.679298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.679278 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:20.694964 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.694935 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:39:20.698471 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.698447 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7"] Oct 31 17:39:20.704925 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.704900 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:39:20.805865 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.805824 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05dc4675-jlms7\" (UID: \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:20.806040 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.805974 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05dc4675-jlms7\" (UID: \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:20.806040 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.806023 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwmjj\" (UniqueName: \"kubernetes.io/projected/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-kube-api-access-fwmjj\") pod \"maas-api-revoke-sub-05dc4675-jlms7\" (UID: \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:20.907370 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.907336 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05dc4675-jlms7\" (UID: \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:20.907557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.907384 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwmjj\" (UniqueName: \"kubernetes.io/projected/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-kube-api-access-fwmjj\") pod \"maas-api-revoke-sub-05dc4675-jlms7\" (UID: \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:20.907557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.907521 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05dc4675-jlms7\" (UID: \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:20.908437 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.908407 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05dc4675-jlms7\" (UID: \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:20.909812 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.909790 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05dc4675-jlms7\" (UID: \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:20.933326 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.930962 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwmjj\" (UniqueName: \"kubernetes.io/projected/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-kube-api-access-fwmjj\") pod \"maas-api-revoke-sub-05dc4675-jlms7\" (UID: \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:20.992697 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:20.992661 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:21.142275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:21.139656 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 17:39:21.142275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:21.139721 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" podUID="446910ca-78af-4c6f-8196-56c209b33636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 17:39:21.347327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:21.347290 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7"] Oct 31 17:39:21.351320 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:39:21.351289 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51abcbf_e4ec_487f_8c4d_d239cc25a0a2.slice/crio-b265f39205e9b0c1a77e34f626d4b24f240b794a63ee7a2d07a70a6cf984db6a WatchSource:0}: Error finding container b265f39205e9b0c1a77e34f626d4b24f240b794a63ee7a2d07a70a6cf984db6a: Status 404 returned error can't find the container with id b265f39205e9b0c1a77e34f626d4b24f240b794a63ee7a2d07a70a6cf984db6a Oct 31 17:39:22.121878 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:22.121779 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 17:39:22.122481 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:22.122445 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" podUID="446910ca-78af-4c6f-8196-56c209b33636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 17:39:22.363471 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:22.363429 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" event={"ID":"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2","Type":"ContainerStarted","Data":"2e76e4ceaa439833d8d1fc48df718fa2314c2155ed63969d4611f31ae3f03070"} Oct 31 17:39:22.363655 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:22.363560 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" event={"ID":"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2","Type":"ContainerStarted","Data":"b265f39205e9b0c1a77e34f626d4b24f240b794a63ee7a2d07a70a6cf984db6a"} Oct 31 17:39:22.399798 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:22.399745 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" podStartSLOduration=2.399729751 podStartE2EDuration="2.399729751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:22.396741952 +0000 UTC m=+684.144716423" watchObservedRunningTime="2026-10-31 17:39:22.399729751 +0000 UTC m=+684.147704217" Oct 31 17:39:23.128736 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:23.128696 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 17:39:23.129348 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:23.129311 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" podUID="446910ca-78af-4c6f-8196-56c209b33636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 17:39:23.371117 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:23.371075 7618 generic.go:356] "Generic (PLEG): container finished" podID="f51abcbf-e4ec-487f-8c4d-d239cc25a0a2" containerID="2e76e4ceaa439833d8d1fc48df718fa2314c2155ed63969d4611f31ae3f03070" exitCode=0 Oct 31 17:39:23.371310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:23.371233 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" event={"ID":"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2","Type":"ContainerDied","Data":"2e76e4ceaa439833d8d1fc48df718fa2314c2155ed63969d4611f31ae3f03070"} Oct 31 17:39:24.121681 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:24.121638 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 17:39:24.121858 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:24.121719 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" podUID="446910ca-78af-4c6f-8196-56c209b33636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 17:39:24.606707 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:24.606674 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:24.787782 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:24.787748 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-kube-api-access-fwmjj\" (UniqueName: \"kubernetes.io/projected/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-kube-api-access-fwmjj\") pod \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\" (UID: \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\") " Oct 31 17:39:24.787978 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:24.787797 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-xks-ca\") pod \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\" (UID: \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\") " Oct 31 17:39:24.787978 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:24.787866 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-openshift-ca\") pod \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\" (UID: \"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2\") " Oct 31 17:39:24.788338 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:24.788305 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-openshift-ca" pod "f51abcbf-e4ec-487f-8c4d-d239cc25a0a2" (UID: "f51abcbf-e4ec-487f-8c4d-d239cc25a0a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:39:24.790101 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:24.790064 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-kube-api-access-fwmjj" pod "f51abcbf-e4ec-487f-8c4d-d239cc25a0a2" (UID: "f51abcbf-e4ec-487f-8c4d-d239cc25a0a2"). InnerVolumeSpecName "kube-api-access-fwmjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:39:24.790196 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:24.790157 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-xks-ca" pod "f51abcbf-e4ec-487f-8c4d-d239cc25a0a2" (UID: "f51abcbf-e4ec-487f-8c4d-d239cc25a0a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:39:24.889335 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:24.889294 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwmjj\" (UniqueName: \"kubernetes.io/projected/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-kube-api-access-fwmjj\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:24.889335 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:24.889329 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:24.889335 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:24.889343 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:25.126191 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:25.126096 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:25.126515 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:25.126488 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:25.127352 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:25.127326 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:39:25.394097 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:25.394069 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" Oct 31 17:39:25.394301 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:25.394106 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7" event={"ID":"f51abcbf-e4ec-487f-8c4d-d239cc25a0a2","Type":"ContainerDied","Data":"b265f39205e9b0c1a77e34f626d4b24f240b794a63ee7a2d07a70a6cf984db6a"} Oct 31 17:39:25.394301 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:25.394150 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b265f39205e9b0c1a77e34f626d4b24f240b794a63ee7a2d07a70a6cf984db6a" Oct 31 17:39:27.841604 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.841553 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq"] Oct 31 17:39:27.842380 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.842358 7618 cpu_manager.go:414] "containerMap: removing container" podUID="f51abcbf-e4ec-487f-8c4d-d239cc25a0a2" containerName="revoke-keys" Oct 31 17:39:27.842434 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.842383 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51abcbf-e4ec-487f-8c4d-d239cc25a0a2" containerName="revoke-keys" Oct 31 17:39:27.842519 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.842506 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51abcbf-e4ec-487f-8c4d-d239cc25a0a2" containerName="revoke-keys" Oct 31 17:39:27.846389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.846356 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:27.870348 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.870313 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq"] Oct 31 17:39:27.920980 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.920936 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:27.920980 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.920980 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:27.921215 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.921094 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f04cec43-eec4-4872-8f9b-8609fc89a551-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:27.921215 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.921134 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04cec43-eec4-4872-8f9b-8609fc89a551-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:27.921215 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.921166 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:27.921411 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.921243 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:27.921411 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.921289 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8lg\" (UniqueName: \"kubernetes.io/projected/f04cec43-eec4-4872-8f9b-8609fc89a551-kube-api-access-lw8lg\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:27.921411 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.921335 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:27.921611 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:27.921468 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04cec43-eec4-4872-8f9b-8609fc89a551-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.022494 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.022454 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f04cec43-eec4-4872-8f9b-8609fc89a551-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.022649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.022503 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04cec43-eec4-4872-8f9b-8609fc89a551-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.022649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.022540 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.022649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.022579 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.022649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.022603 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8lg\" (UniqueName: \"kubernetes.io/projected/f04cec43-eec4-4872-8f9b-8609fc89a551-kube-api-access-lw8lg\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.022649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.022630 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.022924 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.022693 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04cec43-eec4-4872-8f9b-8609fc89a551-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.022924 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.022808 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.022924 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.022842 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.023145 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.023118 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.023234 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.023180 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.023234 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.023211 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.023348 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.023301 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.023581 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.023559 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04cec43-eec4-4872-8f9b-8609fc89a551-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.025064 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.025044 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04cec43-eec4-4872-8f9b-8609fc89a551-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.025390 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.025374 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04cec43-eec4-4872-8f9b-8609fc89a551-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.039055 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.039018 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f04cec43-eec4-4872-8f9b-8609fc89a551-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.039187 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.039167 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8lg\" (UniqueName: \"kubernetes.io/projected/f04cec43-eec4-4872-8f9b-8609fc89a551-kube-api-access-lw8lg\") pod \"maas-default-gateway-openshift-default-7d9897b54c-hk9mq\" (UID: \"f04cec43-eec4-4872-8f9b-8609fc89a551\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.162054 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.162016 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:28.480189 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.480108 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq"] Oct 31 17:39:28.483178 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:39:28.483148 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04cec43_eec4_4872_8f9b_8609fc89a551.slice/crio-c446f0d5868a3e6f546eb5e03c7c722b47e13ff8b743661f40de20c2a23fd61c WatchSource:0}: Error finding container c446f0d5868a3e6f546eb5e03c7c722b47e13ff8b743661f40de20c2a23fd61c: Status 404 returned error can't find the container with id c446f0d5868a3e6f546eb5e03c7c722b47e13ff8b743661f40de20c2a23fd61c Oct 31 17:39:28.485961 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.485923 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:28.486042 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.486015 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:28.486078 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.486056 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:28.987267 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.987225 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h"] Oct 31 17:39:28.991334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:28.991283 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:29.007574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.007537 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h"] Oct 31 17:39:29.014779 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.014752 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:39:29.014925 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.014851 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:39:29.035281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.035235 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b72c1ebb-xxs9h\" (UID: \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:29.035484 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.035305 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49clr\" (UniqueName: \"kubernetes.io/projected/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-kube-api-access-49clr\") pod \"maas-api-revoke-sub-b72c1ebb-xxs9h\" (UID: \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:29.035484 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.035452 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b72c1ebb-xxs9h\" (UID: \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:29.136421 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.136379 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b72c1ebb-xxs9h\" (UID: \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:29.136575 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.136534 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b72c1ebb-xxs9h\" (UID: \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:29.136642 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.136590 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49clr\" (UniqueName: \"kubernetes.io/projected/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-kube-api-access-49clr\") pod \"maas-api-revoke-sub-b72c1ebb-xxs9h\" (UID: \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:29.137111 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.137082 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b72c1ebb-xxs9h\" (UID: \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:29.141825 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.141800 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b72c1ebb-xxs9h\" (UID: \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:29.153771 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.153743 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49clr\" (UniqueName: \"kubernetes.io/projected/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-kube-api-access-49clr\") pod \"maas-api-revoke-sub-b72c1ebb-xxs9h\" (UID: \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:29.305750 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.305669 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:29.427076 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.427031 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" event={"ID":"f04cec43-eec4-4872-8f9b-8609fc89a551","Type":"ContainerStarted","Data":"9a40c496664b8fd229d18f04a1372eaeac344ad86f50c27093a6e91c81b0affd"} Oct 31 17:39:29.427315 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.427109 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" event={"ID":"f04cec43-eec4-4872-8f9b-8609fc89a551","Type":"ContainerStarted","Data":"c446f0d5868a3e6f546eb5e03c7c722b47e13ff8b743661f40de20c2a23fd61c"} Oct 31 17:39:29.466142 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.466071 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podStartSLOduration=2.46605057 podStartE2EDuration="2.46605057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:29.465099479 +0000 UTC m=+691.213073948" watchObservedRunningTime="2026-10-31 17:39:29.46605057 +0000 UTC m=+691.214025038" Oct 31 17:39:29.622350 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:29.621487 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h"] Oct 31 17:39:30.162270 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:30.162211 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:30.164121 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:30.164087 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 17:39:30.164244 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:30.164155 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 17:39:30.457512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:30.457462 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" event={"ID":"4de7eef7-c55f-4da6-be7f-9f41daa2efd3","Type":"ContainerStarted","Data":"cd1572290d0e02d98b29ee7f367c48950963d8696739c671f38a9ddaceafa43b"} Oct 31 17:39:30.457798 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:30.457771 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" event={"ID":"4de7eef7-c55f-4da6-be7f-9f41daa2efd3","Type":"ContainerStarted","Data":"ee766ba12b2004f02b507a2c76901df75bd156f0459aa015be8037d2ecd4f27b"} Oct 31 17:39:30.492796 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:30.492728 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" podStartSLOduration=2.492712599 podStartE2EDuration="2.492712599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:30.48996526 +0000 UTC m=+692.237939711" watchObservedRunningTime="2026-10-31 17:39:30.492712599 +0000 UTC m=+692.240687143" Oct 31 17:39:31.156495 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.156457 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc"] Oct 31 17:39:31.161600 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.161571 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.163675 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.163646 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 17:39:31.164170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.164141 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 17:39:31.186342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.186311 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-cf0a86-openshift-default-dockercfg-m2mbq\"" Oct 31 17:39:31.207606 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.207561 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc"] Oct 31 17:39:31.278583 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.278545 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-token\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.278899 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.278876 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-certs\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.279062 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.279043 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-podinfo\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.279204 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.279182 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.279413 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.279393 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istiod-ca-cert\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.279539 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.279526 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh8n9\" (UniqueName: \"kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-kube-api-access-lh8n9\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.279729 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.279714 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-credential-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.279831 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.279818 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-data\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.279947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.279935 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-envoy\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.386376 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.386340 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.386579 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.386431 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istiod-ca-cert\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.386579 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.386478 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8n9\" (UniqueName: \"kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-kube-api-access-lh8n9\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.386702 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.386600 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-credential-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.386702 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.386628 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-data\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.386702 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.386670 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-envoy\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.386702 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.386698 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-token\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.386917 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.386734 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-certs\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.386917 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.386777 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-podinfo\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.388299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.387452 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-credential-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.388299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.387983 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istiod-ca-cert\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.388299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.388023 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.388299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.388300 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-data\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.388714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.388696 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-certs\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.391165 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.391140 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-envoy\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.391894 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.391872 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-podinfo\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.406739 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.405550 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-token\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.406991 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.406966 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh8n9\" (UniqueName: \"kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-kube-api-access-lh8n9\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.467286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.465773 7618 generic.go:356] "Generic (PLEG): container finished" podID="4de7eef7-c55f-4da6-be7f-9f41daa2efd3" containerID="cd1572290d0e02d98b29ee7f367c48950963d8696739c671f38a9ddaceafa43b" exitCode=0 Oct 31 17:39:31.467286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.465873 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" event={"ID":"4de7eef7-c55f-4da6-be7f-9f41daa2efd3","Type":"ContainerDied","Data":"cd1572290d0e02d98b29ee7f367c48950963d8696739c671f38a9ddaceafa43b"} Oct 31 17:39:31.480310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.480283 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:31.665328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.662908 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l"] Oct 31 17:39:31.672272 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.669749 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l"] Oct 31 17:39:31.672272 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.669862 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.686276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.683493 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-d80080-openshift-default-dockercfg-hqzw7\"" Oct 31 17:39:31.773277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.772297 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht"] Oct 31 17:39:31.794559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.793890 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdg4r\" (UniqueName: \"kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-kube-api-access-bdg4r\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.794559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.793994 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-podinfo\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.794559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.794051 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.794559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.794116 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istiod-ca-cert\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.794559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.794169 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-credential-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.794559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.794198 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-envoy\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.794559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.794231 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-data\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.794559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.794294 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-certs\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.794559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.794325 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-token\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.811866 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.811835 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht"] Oct 31 17:39:31.812053 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.811961 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:31.860274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.831267 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5c446c-openshift-default-dockercfg-kfjdh\"" Oct 31 17:39:31.871743 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:39:31.871708 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08f2d28_721a_4f5c_9e8e_db5aefafdd0a.slice/crio-a40828529206ae06e4cefd52ad194f7dc90ac088dd83577efe51c008df8cf6e2 WatchSource:0}: Error finding container a40828529206ae06e4cefd52ad194f7dc90ac088dd83577efe51c008df8cf6e2: Status 404 returned error can't find the container with id a40828529206ae06e4cefd52ad194f7dc90ac088dd83577efe51c008df8cf6e2 Oct 31 17:39:31.876137 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.876083 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:31.876269 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.876202 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:31.876343 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.876305 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:31.894185 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.894153 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc"] Oct 31 17:39:31.895497 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.895463 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-podinfo\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.895623 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.895526 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-credential-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:31.895623 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.895566 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.895736 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.895685 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istiod-ca-cert\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.895855 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.895733 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-podinfo\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:31.895855 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.895788 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-credential-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.895855 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.895818 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-envoy\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.895855 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.895848 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-data\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.896071 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.895878 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-certs\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:31.896071 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.895919 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-certs\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.896071 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.895949 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-token\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.896071 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.895977 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-token\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:31.896071 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.896009 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf88942-00cd-41b1-9a86-9591b5e41283-istiod-ca-cert\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:31.896649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.896075 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w27kd\" (UniqueName: \"kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-kube-api-access-w27kd\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:31.896649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.896122 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdg4r\" (UniqueName: \"kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-kube-api-access-bdg4r\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.896649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.896190 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:31.896649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.896213 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-data\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:31.896649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.896264 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-envoy\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:31.897199 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.896673 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.897199 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.896947 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-certs\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.897199 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.897185 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-data\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.897477 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.897435 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-credential-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.898074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.898048 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istiod-ca-cert\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.900125 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.900082 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-envoy\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.901600 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.901573 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-podinfo\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.914498 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.914432 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdg4r\" (UniqueName: \"kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-kube-api-access-bdg4r\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.915882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.915795 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-token\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:31.993681 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:31.993630 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4"] Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.002822 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.003755 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.003799 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-data\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.003829 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-envoy\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.003870 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-credential-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.003941 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-podinfo\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.004014 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-certs\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.004079 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-token\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.004108 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf88942-00cd-41b1-9a86-9591b5e41283-istiod-ca-cert\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.004165 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w27kd\" (UniqueName: \"kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-kube-api-access-w27kd\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.005014 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.005280 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-data\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.005832 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-certs\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.006051 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-credential-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.010840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.010762 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf88942-00cd-41b1-9a86-9591b5e41283-istiod-ca-cert\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.105523 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.012906 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.105523 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.014860 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-envoy\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.105523 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.045919 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-podinfo\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.105523 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.047663 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-0e59ca-openshift-default-dockercfg-75hsv\"" Oct 31 17:39:32.105523 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.059046 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w27kd\" (UniqueName: \"kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-kube-api-access-w27kd\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.116400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.116304 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4"] Oct 31 17:39:32.124828 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.124759 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-token\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.134729 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.134468 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:32.170783 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.170628 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 17:39:32.171361 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.170700 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 17:39:32.208045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.207070 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-data\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.208045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.207137 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-credential-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.208045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.207184 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb83a702-5210-41d1-b099-1c7d515c99d7-istiod-ca-cert\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.208045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.207286 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.208045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.207323 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq2l2\" (UniqueName: \"kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-kube-api-access-bq2l2\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.208045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.207375 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-certs\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.208045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.207427 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-envoy\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.208045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.207546 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-podinfo\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.208045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.207569 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-token\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.333004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.331322 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.333004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.331378 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq2l2\" (UniqueName: \"kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-kube-api-access-bq2l2\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.333004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.331430 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-certs\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.333004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.331478 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-envoy\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.333004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.331593 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-podinfo\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.333004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.331619 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-token\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.333004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.331689 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-data\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.333004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.331733 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-credential-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.333004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.331777 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb83a702-5210-41d1-b099-1c7d515c99d7-istiod-ca-cert\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.333004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.332626 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb83a702-5210-41d1-b099-1c7d515c99d7-istiod-ca-cert\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.333004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.332980 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-data\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.333787 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.333763 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.334045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.334018 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-certs\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.340273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.334354 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-credential-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.404272 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.403926 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-podinfo\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.443308 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.442643 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq2l2\" (UniqueName: \"kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-kube-api-access-bq2l2\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.443308 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.443074 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-token\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.443308 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.443213 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-envoy\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.494426 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.494310 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" event={"ID":"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a","Type":"ContainerStarted","Data":"c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05"} Oct 31 17:39:32.494426 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.494380 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" event={"ID":"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a","Type":"ContainerStarted","Data":"a40828529206ae06e4cefd52ad194f7dc90ac088dd83577efe51c008df8cf6e2"} Oct 31 17:39:32.587630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.587558 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podStartSLOduration=1.587536499 podStartE2EDuration="1.587536499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:32.567489295 +0000 UTC m=+694.315463787" watchObservedRunningTime="2026-10-31 17:39:32.587536499 +0000 UTC m=+694.335510967" Oct 31 17:39:32.703558 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.703523 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:32.970197 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.967311 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l"] Oct 31 17:39:32.981429 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.980989 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht"] Oct 31 17:39:32.983071 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.983038 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:32.983179 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.983102 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:32.983179 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.983133 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:32.983527 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.983511 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:32.989687 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.988647 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:32.989687 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.988735 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:32.989687 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:32.988785 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:33.100651 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.100618 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-openshift-ca\") pod \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\" (UID: \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\") " Oct 31 17:39:33.100782 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.100761 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-kube-api-access-49clr\" (UniqueName: \"kubernetes.io/projected/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-kube-api-access-49clr\") pod \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\" (UID: \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\") " Oct 31 17:39:33.101453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.100853 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-xks-ca\") pod \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\" (UID: \"4de7eef7-c55f-4da6-be7f-9f41daa2efd3\") " Oct 31 17:39:33.101453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.101137 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-openshift-ca" pod "4de7eef7-c55f-4da6-be7f-9f41daa2efd3" (UID: "4de7eef7-c55f-4da6-be7f-9f41daa2efd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:39:33.101453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.101386 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:33.103427 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.103395 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-kube-api-access-49clr" pod "4de7eef7-c55f-4da6-be7f-9f41daa2efd3" (UID: "4de7eef7-c55f-4da6-be7f-9f41daa2efd3"). InnerVolumeSpecName "kube-api-access-49clr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:39:33.103427 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.103412 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-xks-ca" pod "4de7eef7-c55f-4da6-be7f-9f41daa2efd3" (UID: "4de7eef7-c55f-4da6-be7f-9f41daa2efd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:39:33.164112 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.164080 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 17:39:33.164289 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.164126 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 17:39:33.193819 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.192182 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4"] Oct 31 17:39:33.196097 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:39:33.196051 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb83a702_5210_41d1_b099_1c7d515c99d7.slice/crio-84024fb9d807180391d8b9d44c2545ebfcdfa8b89dec728d4aef75f337e32594 WatchSource:0}: Error finding container 84024fb9d807180391d8b9d44c2545ebfcdfa8b89dec728d4aef75f337e32594: Status 404 returned error can't find the container with id 84024fb9d807180391d8b9d44c2545ebfcdfa8b89dec728d4aef75f337e32594 Oct 31 17:39:33.200342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.200306 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:33.200574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.200535 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:33.200674 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.200592 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:39:33.203436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.203417 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49clr\" (UniqueName: \"kubernetes.io/projected/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-kube-api-access-49clr\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:33.203533 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.203441 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de7eef7-c55f-4da6-be7f-9f41daa2efd3-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:39:33.496627 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.493785 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:33.498312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.496932 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 17:39:33.498312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.496990 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 17:39:33.533702 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.533666 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:39:33.591450 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.583286 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" event={"ID":"cafa0366-2f1f-4814-b8be-43a4d5540e4c","Type":"ContainerStarted","Data":"933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362"} Oct 31 17:39:33.591760 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.591724 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" event={"ID":"cafa0366-2f1f-4814-b8be-43a4d5540e4c","Type":"ContainerStarted","Data":"7a561b08167f69196f0509b831453179b01ff7cd8d963504c9f4129ae958f2eb"} Oct 31 17:39:33.655590 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.645742 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podStartSLOduration=2.645719942 podStartE2EDuration="2.645719942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:33.643821223 +0000 UTC m=+695.391795704" watchObservedRunningTime="2026-10-31 17:39:33.645719942 +0000 UTC m=+695.393694410" Oct 31 17:39:33.680275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.675999 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" event={"ID":"5cf88942-00cd-41b1-9a86-9591b5e41283","Type":"ContainerStarted","Data":"25b33c0ea918c5c296923cb846b4001154c30b29bff7679599c59150e4ad6f70"} Oct 31 17:39:33.680275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.676076 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" event={"ID":"5cf88942-00cd-41b1-9a86-9591b5e41283","Type":"ContainerStarted","Data":"5423315ad5e7642074baf7b0f15f78c0dc90f51bd1620d1e21a458d4d5937ae2"} Oct 31 17:39:33.737281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.717402 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" event={"ID":"fb83a702-5210-41d1-b099-1c7d515c99d7","Type":"ContainerStarted","Data":"e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0"} Oct 31 17:39:33.737281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.717483 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" event={"ID":"fb83a702-5210-41d1-b099-1c7d515c99d7","Type":"ContainerStarted","Data":"84024fb9d807180391d8b9d44c2545ebfcdfa8b89dec728d4aef75f337e32594"} Oct 31 17:39:33.737281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.731500 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podStartSLOduration=2.731482058 podStartE2EDuration="2.731482058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:33.729390161 +0000 UTC m=+695.477364655" watchObservedRunningTime="2026-10-31 17:39:33.731482058 +0000 UTC m=+695.479456526" Oct 31 17:39:33.740939 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.740910 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" Oct 31 17:39:33.756748 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.741148 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h" event={"ID":"4de7eef7-c55f-4da6-be7f-9f41daa2efd3","Type":"ContainerDied","Data":"ee766ba12b2004f02b507a2c76901df75bd156f0459aa015be8037d2ecd4f27b"} Oct 31 17:39:33.757015 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.756991 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee766ba12b2004f02b507a2c76901df75bd156f0459aa015be8037d2ecd4f27b" Oct 31 17:39:33.809322 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:33.809237 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podStartSLOduration=2.809193446 podStartE2EDuration="2.809193446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:33.786690116 +0000 UTC m=+695.534664586" watchObservedRunningTime="2026-10-31 17:39:33.809193446 +0000 UTC m=+695.557167914" Oct 31 17:39:34.060541 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.059864 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:34.060541 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.060285 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 17:39:34.060541 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.060341 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 17:39:34.160227 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.159414 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:34.160227 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.159601 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 17:39:34.160227 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.159646 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 17:39:34.165757 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.165714 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 17:39:34.166038 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.165998 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 17:39:34.486676 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.486637 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 17:39:34.487756 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.487721 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 17:39:34.715074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.715032 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:34.715513 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.715482 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 17:39:34.715709 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:34.715679 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 17:39:35.024279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.011814 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 17:39:35.024279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.011894 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 17:39:35.151409 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.144720 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 17:39:35.151409 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.144783 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 17:39:35.166178 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.166139 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 17:39:35.166491 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.166455 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 17:39:35.263408 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.263357 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:35.379471 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.379350 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:35.483414 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.483362 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:35.484549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.484047 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 17:39:35.484549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.484498 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 17:39:35.507397 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.507350 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:35.709737 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.709693 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 17:39:35.710779 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:35.710746 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 17:39:36.006476 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.003718 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 17:39:36.006476 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.003789 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 17:39:36.143712 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.143668 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 17:39:36.143978 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.143940 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 17:39:36.164488 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.164444 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 17:39:36.164801 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.164758 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 17:39:36.276342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.267391 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:36.276342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.268300 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:39:36.485974 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.485918 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:36.552336 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.491377 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 17:39:36.552336 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.549958 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 17:39:36.552336 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.550401 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:39:36.626278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.617462 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:36.733276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.721758 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 17:39:36.733276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.721821 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 17:39:36.785281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:36.766370 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:37.007278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.005644 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 17:39:37.007278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.005710 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 17:39:37.159274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.142665 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 17:39:37.159274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.142734 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 17:39:37.190274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.189597 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 17:39:37.190274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.189657 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 17:39:37.288376 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.287571 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:37.288376 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.288292 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:37.347275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.343681 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 17:39:37.392280 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.388348 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 17:39:37.485275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.481700 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 17:39:37.485275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.481774 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 17:39:37.510277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.506873 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:37.564331 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.558486 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:37.625293 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.622350 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 31 17:39:37.750286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.725638 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 17:39:37.750286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.725701 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 17:39:37.767398 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.767348 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:37.768109 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:37.768084 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:39:38.041278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.040891 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 17:39:38.041278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.040952 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 17:39:38.093276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.082406 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:38.161277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.154657 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 17:39:38.161277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.154731 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 17:39:38.178226 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.178102 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 17:39:38.178226 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.178165 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 17:39:38.290439 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.290375 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:38.291425 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.291348 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:38.296406 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.296371 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:38.413287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.407418 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:38.510468 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.510410 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:38.560330 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.559323 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:38.626677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.591192 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 17:39:38.626677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.591266 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 17:39:38.644327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.628456 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:39:38.644327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.628546 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:39:38.746282 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.742554 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 17:39:38.746282 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.742612 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 17:39:38.775873 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:38.775401 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 17:39:39.017288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.005042 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 17:39:39.102349 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.102305 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 17:39:39.176280 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.174400 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 17:39:39.176280 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.174468 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 17:39:39.176280 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.176230 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 17:39:39.176609 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.176298 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 17:39:39.206162 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.206124 7618 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 17:39:39.300813 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.300311 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:39.301209 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.301157 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:39.329571 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:39:39.329498 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:39:39.358752 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.358704 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:39.486384 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.486347 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 17:39:39.486664 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.486628 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 17:39:39.567605 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.567494 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:39.568005 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.567972 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:39.589106 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.589020 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:39.629410 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.629363 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:39.704278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.703362 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" podUID="60303a6b-433f-4b1c-8e2e-19e901fbd933" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:39.705561 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.705530 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 17:39:39.705824 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.705776 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 17:39:39.905152 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:39.905094 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:40.013645 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.013608 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:39:40.015167 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.015135 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 17:39:40.015397 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.015369 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 17:39:40.158331 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.156176 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 17:39:40.158331 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.156237 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 17:39:40.307540 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.307346 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 17:39:40.309970 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.307790 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 17:39:40.359400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.359350 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:40.360345 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.360320 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:39:40.484780 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.484684 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 17:39:40.485035 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.485004 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 17:39:40.494677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.494636 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:40.568645 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.568598 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:40.587021 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.570179 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 17:39:40.631737 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.631696 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:40.656277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.633901 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:39:40.775276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.767424 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 17:39:40.775276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.767489 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 17:39:40.907533 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:40.906342 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:41.006489 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.006043 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 17:39:41.006489 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.006107 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 17:39:41.144689 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.144651 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 17:39:41.144935 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.144911 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 17:39:41.210841 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.210792 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:39:41.211074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.211050 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:41.310752 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.310694 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:41.311525 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.311035 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:41.364900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.364855 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:41.391967 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.391930 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:39:41.410571 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.410484 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:39:41.507762 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.507713 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 17:39:41.507956 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.507801 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 17:39:41.570574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.570535 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:41.587768 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.587732 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:41.634407 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.633332 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:41.705128 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.704649 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 17:39:41.705128 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.704710 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 17:39:41.751795 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.751748 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:39:41.867738 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.866741 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:39:41.912299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.912224 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:41.939615 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.939581 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:39:41.991799 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:41.991719 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:39:42.081309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.069536 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 17:39:42.081309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.069590 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 17:39:42.153284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.151354 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 17:39:42.153284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.151414 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 17:39:42.256534 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.256428 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:39:42.256824 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.256776 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:42.371275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.370980 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:39:42.428824 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.428789 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:39:42.466276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.465438 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:42.470273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.466815 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:42.470273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.469144 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" Oct 31 17:39:42.486275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.481562 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 17:39:42.486275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.481634 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 17:39:42.514885 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.512618 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 17:39:42.705908 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.705857 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 17:39:42.706536 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:42.706488 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 17:39:43.004280 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:43.004205 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 17:39:43.004494 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:43.004292 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 17:39:43.379501 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:43.379354 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 17:39:43.379880 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:43.379752 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 17:39:43.492487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:43.492437 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 17:39:43.513299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:43.513232 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 17:39:43.704736 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:43.704696 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 17:39:43.705039 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:43.705006 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 17:39:44.004326 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:44.003773 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 17:39:44.004326 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:44.003851 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 17:39:44.160130 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:44.160090 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 17:39:44.160473 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:44.160440 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 17:39:44.704187 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:44.704144 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 17:39:44.704418 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:44.704218 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 17:39:45.144277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.142675 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 17:39:45.144277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.142746 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 17:39:45.237613 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.237576 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:45.239386 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.239361 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:45.251573 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.251541 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:39:45.486319 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.485467 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:39:45.486319 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.485554 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:39:45.585275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.584396 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:45.586889 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.586232 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:45.588633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.588609 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:39:45.740332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.725587 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:45.740332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.727047 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:45.740332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:45.730389 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:39:46.794375 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:46.794335 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:46.794869 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:46.794791 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:46.796040 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:46.796017 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:39:48.956869 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:48.956835 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f"] Oct 31 17:39:48.959085 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:48.959055 7618 cpu_manager.go:414] "containerMap: removing container" podUID="4de7eef7-c55f-4da6-be7f-9f41daa2efd3" containerName="revoke-keys" Oct 31 17:39:48.959085 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:48.959084 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de7eef7-c55f-4da6-be7f-9f41daa2efd3" containerName="revoke-keys" Oct 31 17:39:48.959304 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:48.959277 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de7eef7-c55f-4da6-be7f-9f41daa2efd3" containerName="revoke-keys" Oct 31 17:39:48.962977 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:48.962950 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:48.991160 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:48.991116 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f"] Oct 31 17:39:48.991415 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:48.991392 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0e59ca\"/\"distinct-2-w1-0e59ca-kserve-self-signed-certs\"" Oct 31 17:39:49.102864 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.102825 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-home\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.103047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.103016 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m77qx\" (UniqueName: \"kubernetes.io/projected/baeaac10-b796-4dcd-9d24-752c346dff6a-kube-api-access-m77qx\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.103140 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.103123 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-tmp-dir\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.103218 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.103203 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-model-cache\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.103315 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.103300 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-dshm\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.103374 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.103359 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baeaac10-b796-4dcd-9d24-752c346dff6a-tls-certs\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.205108 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.205054 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m77qx\" (UniqueName: \"kubernetes.io/projected/baeaac10-b796-4dcd-9d24-752c346dff6a-kube-api-access-m77qx\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.205339 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.205152 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-tmp-dir\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.205339 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.205213 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-model-cache\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.205339 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.205276 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-dshm\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.205339 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.205335 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baeaac10-b796-4dcd-9d24-752c346dff6a-tls-certs\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.205557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.205393 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-home\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.205969 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.205941 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-home\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.206223 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.206205 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-model-cache\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.206386 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.206220 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-tmp-dir\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.229732 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.229649 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baeaac10-b796-4dcd-9d24-752c346dff6a-tls-certs\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.230156 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.230124 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-dshm\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.234036 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.234005 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m77qx\" (UniqueName: \"kubernetes.io/projected/baeaac10-b796-4dcd-9d24-752c346dff6a-kube-api-access-m77qx\") pod \"distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.281473 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.281435 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:49.597324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.597298 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f"] Oct 31 17:39:49.599863 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:39:49.599821 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaeaac10_b796_4dcd_9d24_752c346dff6a.slice/crio-46b063fe10b6b16a47609250ddc611e138e94c3e1db4a5af2ee4212efd7c3000 WatchSource:0}: Error finding container 46b063fe10b6b16a47609250ddc611e138e94c3e1db4a5af2ee4212efd7c3000: Status 404 returned error can't find the container with id 46b063fe10b6b16a47609250ddc611e138e94c3e1db4a5af2ee4212efd7c3000 Oct 31 17:39:49.630389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:49.628884 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" event={"ID":"baeaac10-b796-4dcd-9d24-752c346dff6a","Type":"ContainerStarted","Data":"46b063fe10b6b16a47609250ddc611e138e94c3e1db4a5af2ee4212efd7c3000"} Oct 31 17:39:50.639314 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:50.639266 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" event={"ID":"baeaac10-b796-4dcd-9d24-752c346dff6a","Type":"ContainerStarted","Data":"f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940"} Oct 31 17:39:50.675230 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:50.675174 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podStartSLOduration=2.675159253 podStartE2EDuration="2.675159253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:39:50.672742423 +0000 UTC m=+712.420716891" watchObservedRunningTime="2026-10-31 17:39:50.675159253 +0000 UTC m=+712.423133760" Oct 31 17:39:51.281792 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:51.281754 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:59.281809 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:59.281771 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:59.295109 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:59.295078 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:39:59.733424 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:39:59.733392 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:40:00.594981 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.594946 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8"] Oct 31 17:40:00.600673 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.600642 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.619523 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.619484 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8"] Oct 31 17:40:00.624106 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.624081 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cf0a86\"/\"distinct-w3-cf0a86-kserve-self-signed-certs\"" Oct 31 17:40:00.781670 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.781633 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-model-cache\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.781892 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.781807 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cacd0841-f34a-4ad3-8534-e13306ad2037-tls-certs\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.781892 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.781854 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-dshm\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.782014 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.781906 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-tmp-dir\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.782014 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.781936 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-home\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.782014 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.781999 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhdn\" (UniqueName: \"kubernetes.io/projected/cacd0841-f34a-4ad3-8534-e13306ad2037-kube-api-access-glhdn\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.883246 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.883137 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-model-cache\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.883246 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.883238 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cacd0841-f34a-4ad3-8534-e13306ad2037-tls-certs\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.883496 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.883278 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-dshm\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.883496 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.883322 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-tmp-dir\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.883496 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.883353 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-home\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.883496 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.883428 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhdn\" (UniqueName: \"kubernetes.io/projected/cacd0841-f34a-4ad3-8534-e13306ad2037-kube-api-access-glhdn\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.884439 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.884369 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-home\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.884669 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.884647 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-tmp-dir\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.884744 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.884728 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-model-cache\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.886770 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.886745 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-dshm\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.887863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.887844 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cacd0841-f34a-4ad3-8534-e13306ad2037-tls-certs\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.902502 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.902471 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhdn\" (UniqueName: \"kubernetes.io/projected/cacd0841-f34a-4ad3-8534-e13306ad2037-kube-api-access-glhdn\") pod \"distinct-w3-cf0a86-kserve-d764fddbb-nczg8\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:00.923558 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:00.923520 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:01.259132 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:01.259096 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8"] Oct 31 17:40:01.267084 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:40:01.265753 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcacd0841_f34a_4ad3_8534_e13306ad2037.slice/crio-349b6c73406ce6e063afe466872ff9ad02c41130ba831600cdb7b5b7c2f444b8 WatchSource:0}: Error finding container 349b6c73406ce6e063afe466872ff9ad02c41130ba831600cdb7b5b7c2f444b8: Status 404 returned error can't find the container with id 349b6c73406ce6e063afe466872ff9ad02c41130ba831600cdb7b5b7c2f444b8 Oct 31 17:40:01.751512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:01.751469 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" event={"ID":"cacd0841-f34a-4ad3-8534-e13306ad2037","Type":"ContainerStarted","Data":"5f678654ede31354060cb3baf696c6c4bc4a8ab23ef2fbadbd4bd3d49eff676d"} Oct 31 17:40:01.751947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:01.751526 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" event={"ID":"cacd0841-f34a-4ad3-8534-e13306ad2037","Type":"ContainerStarted","Data":"349b6c73406ce6e063afe466872ff9ad02c41130ba831600cdb7b5b7c2f444b8"} Oct 31 17:40:01.788414 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:01.788340 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podStartSLOduration=1.7883264300000001 podStartE2EDuration="1.78832643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:40:01.784651907 +0000 UTC m=+723.532626373" watchObservedRunningTime="2026-10-31 17:40:01.78832643 +0000 UTC m=+723.536300888" Oct 31 17:40:01.925673 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:01.925635 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:03.877518 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:03.877477 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:40:08.574824 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:40:08.574776 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:40:10.926419 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:10.926329 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:10.939139 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:10.939109 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:11.843682 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:11.843648 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:40:16.850396 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:16.850357 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v"] Oct 31 17:40:16.854688 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:16.854665 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:16.890299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:16.888726 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v"] Oct 31 17:40:16.993348 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:16.993307 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-credential-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:16.993562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:16.993409 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpslm\" (UniqueName: \"kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-kube-api-access-vpslm\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:16.993562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:16.993517 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-certs\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:16.993688 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:16.993592 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-envoy\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:16.993754 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:16.993724 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-token\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:16.993811 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:16.993773 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-podinfo\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:16.993864 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:16.993833 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-data\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:16.993920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:16.993866 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac3a54c-5d8f-4451-9584-dec64d6d235c-istiod-ca-cert\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:16.993981 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:16.993934 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.095446 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.095405 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-data\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.095446 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.095460 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac3a54c-5d8f-4451-9584-dec64d6d235c-istiod-ca-cert\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.095740 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.095512 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.095740 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.095544 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-credential-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.095740 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.095571 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpslm\" (UniqueName: \"kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-kube-api-access-vpslm\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.095740 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.095650 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-certs\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.095740 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.095676 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-envoy\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.096000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.095811 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-token\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.096000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.095858 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-podinfo\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.096000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.095862 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-data\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.096149 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.095998 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-credential-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.096149 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.096074 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.096381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.096344 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac3a54c-5d8f-4451-9584-dec64d6d235c-istiod-ca-cert\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.096480 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.096434 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-certs\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.098329 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.098305 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-podinfo\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.098438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.098345 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-envoy\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.113418 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.113349 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-token\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.113541 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.113432 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpslm\" (UniqueName: \"kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-kube-api-access-vpslm\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.187511 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.187469 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:17.522157 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.522110 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v"] Oct 31 17:40:17.534371 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:40:17.534323 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac3a54c_5d8f_4451_9584_dec64d6d235c.slice/crio-928cc4ef305590ed91c20928abeec94294b36c50ee880e166c04288ebd4e038c WatchSource:0}: Error finding container 928cc4ef305590ed91c20928abeec94294b36c50ee880e166c04288ebd4e038c: Status 404 returned error can't find the container with id 928cc4ef305590ed91c20928abeec94294b36c50ee880e166c04288ebd4e038c Oct 31 17:40:17.537438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.537398 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:40:17.537568 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.537504 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:40:17.537568 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.537552 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:40:17.873577 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.873478 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" event={"ID":"bac3a54c-5d8f-4451-9584-dec64d6d235c","Type":"ContainerStarted","Data":"cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1"} Oct 31 17:40:17.873577 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.873548 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" event={"ID":"bac3a54c-5d8f-4451-9584-dec64d6d235c","Type":"ContainerStarted","Data":"928cc4ef305590ed91c20928abeec94294b36c50ee880e166c04288ebd4e038c"} Oct 31 17:40:17.916907 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:17.916856 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" podStartSLOduration=1.9168404749999999 podStartE2EDuration="1.916840475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:40:17.914372238 +0000 UTC m=+739.662346706" watchObservedRunningTime="2026-10-31 17:40:17.916840475 +0000 UTC m=+739.664815274" Oct 31 17:40:18.188326 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:18.188295 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:18.189617 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:18.189586 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 17:40:18.189761 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:18.189641 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 17:40:18.914041 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:18.914001 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr"] Oct 31 17:40:18.954888 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:18.954854 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:18.969474 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:18.969438 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr"] Oct 31 17:40:18.987102 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:18.987070 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d80080\"/\"distinct-2-w6-d80080-kserve-self-signed-certs\"" Oct 31 17:40:19.033222 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.033173 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-model-cache\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.048143 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.048107 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-home\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.048504 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.048480 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvlh\" (UniqueName: \"kubernetes.io/projected/58baa22b-b233-4796-82d2-983a7ec80b0b-kube-api-access-zfvlh\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.048803 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.048782 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-tmp-dir\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.049274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.049237 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-dshm\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.049734 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.049715 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58baa22b-b233-4796-82d2-983a7ec80b0b-tls-certs\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.151281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.151080 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-model-cache\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.151281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.151148 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-home\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.151281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.151183 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvlh\" (UniqueName: \"kubernetes.io/projected/58baa22b-b233-4796-82d2-983a7ec80b0b-kube-api-access-zfvlh\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.151591 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.151314 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-tmp-dir\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.151591 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.151364 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-dshm\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.151591 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.151406 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58baa22b-b233-4796-82d2-983a7ec80b0b-tls-certs\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.152739 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.152709 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-model-cache\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.153196 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.153172 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-home\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.153601 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.153581 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-tmp-dir\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.162951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.160971 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58baa22b-b233-4796-82d2-983a7ec80b0b-tls-certs\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.165921 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.165857 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-dshm\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.179041 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.179008 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvlh\" (UniqueName: \"kubernetes.io/projected/58baa22b-b233-4796-82d2-983a7ec80b0b-kube-api-access-zfvlh\") pod \"distinct-2-w6-d80080-kserve-79788fbcdd-8gldr\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.191299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.189658 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 17:40:19.191299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.189715 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 17:40:19.288596 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.288554 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:19.693048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.692968 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr"] Oct 31 17:40:19.697934 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:40:19.696488 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58baa22b_b233_4796_82d2_983a7ec80b0b.slice/crio-305ee961096a2985a660cabdbb137c72cc59adc28a2b4da0061d347a955982b1 WatchSource:0}: Error finding container 305ee961096a2985a660cabdbb137c72cc59adc28a2b4da0061d347a955982b1: Status 404 returned error can't find the container with id 305ee961096a2985a660cabdbb137c72cc59adc28a2b4da0061d347a955982b1 Oct 31 17:40:19.890005 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.889971 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" event={"ID":"58baa22b-b233-4796-82d2-983a7ec80b0b","Type":"ContainerStarted","Data":"c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c"} Oct 31 17:40:19.890170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:19.890016 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" event={"ID":"58baa22b-b233-4796-82d2-983a7ec80b0b","Type":"ContainerStarted","Data":"305ee961096a2985a660cabdbb137c72cc59adc28a2b4da0061d347a955982b1"} Oct 31 17:40:20.188173 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:20.188137 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 17:40:20.188578 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:20.188205 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 17:40:20.290092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:20.290061 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:21.187754 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:21.187721 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 17:40:21.187970 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:21.187781 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 17:40:22.188506 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:22.188466 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 17:40:22.188922 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:22.188522 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 17:40:23.188275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:23.188218 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 17:40:23.188599 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:23.188552 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 17:40:24.188942 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:24.188891 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 17:40:24.189520 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:24.188965 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 17:40:25.195677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:25.195642 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:25.196110 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:25.196074 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:25.196880 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:25.196856 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:40:25.235902 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:25.235848 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podStartSLOduration=7.2358329 podStartE2EDuration="7.2358329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:40:19.938547244 +0000 UTC m=+741.686521712" watchObservedRunningTime="2026-10-31 17:40:25.2358329 +0000 UTC m=+746.983807367" Oct 31 17:40:26.215484 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.215446 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t"] Oct 31 17:40:26.221155 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.221120 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.242705 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.242485 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t"] Oct 31 17:40:26.244677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.244570 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ee816e\"/\"openshift-service-ca.crt\"" Oct 31 17:40:26.244677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.244570 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee816e\"/\"default-dockercfg-7pq72\"" Oct 31 17:40:26.244677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.244666 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ee816e\"/\"kube-root-ca.crt\"" Oct 31 17:40:26.244954 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.244750 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee816e\"/\"distinct-2-w0-ee816e-kserve-self-signed-certs\"" Oct 31 17:40:26.300834 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.300795 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-home\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.301011 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.300856 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-dshm\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.301011 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.300878 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-model-cache\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.301011 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.300898 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmdkk\" (UniqueName: \"kubernetes.io/projected/f7c13943-5746-4c6e-a4d2-5c11e89156bd-kube-api-access-xmdkk\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.301011 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.300964 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tls-certs\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.301011 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.300985 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tmp-dir\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.402111 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.402066 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-home\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.402327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.402134 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-dshm\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.402327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.402163 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-model-cache\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.402327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.402189 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmdkk\" (UniqueName: \"kubernetes.io/projected/f7c13943-5746-4c6e-a4d2-5c11e89156bd-kube-api-access-xmdkk\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.402509 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.402354 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tls-certs\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.402509 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.402388 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tmp-dir\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.402632 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.402555 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-home\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.402872 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.402848 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tmp-dir\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.403061 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.402859 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-model-cache\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.405740 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.405716 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-dshm\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.406074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.406051 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tls-certs\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.426009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.425981 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmdkk\" (UniqueName: \"kubernetes.io/projected/f7c13943-5746-4c6e-a4d2-5c11e89156bd-kube-api-access-xmdkk\") pod \"distinct-2-w0-ee816e-kserve-f765968c4-9lr2t\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.556120 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.555811 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:26.875316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.875287 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t"] Oct 31 17:40:26.877711 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:40:26.877685 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7c13943_5746_4c6e_a4d2_5c11e89156bd.slice/crio-40fe569f579fa570bcb3108456aad32f98278e68e64c34cb436386b6e39b7fbf WatchSource:0}: Error finding container 40fe569f579fa570bcb3108456aad32f98278e68e64c34cb436386b6e39b7fbf: Status 404 returned error can't find the container with id 40fe569f579fa570bcb3108456aad32f98278e68e64c34cb436386b6e39b7fbf Oct 31 17:40:26.948635 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:26.948600 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" event={"ID":"f7c13943-5746-4c6e-a4d2-5c11e89156bd","Type":"ContainerStarted","Data":"40fe569f579fa570bcb3108456aad32f98278e68e64c34cb436386b6e39b7fbf"} Oct 31 17:40:27.958543 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:27.958500 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" event={"ID":"f7c13943-5746-4c6e-a4d2-5c11e89156bd","Type":"ContainerStarted","Data":"0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d"} Oct 31 17:40:27.996118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:27.996065 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podStartSLOduration=1.996051937 podStartE2EDuration="1.996051937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:40:27.992630817 +0000 UTC m=+749.740605284" watchObservedRunningTime="2026-10-31 17:40:27.996051937 +0000 UTC m=+749.744026404" Oct 31 17:40:28.556689 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:28.556653 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:29.289801 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.289762 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:29.316451 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.316400 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz"] Oct 31 17:40:29.324147 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.324119 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:29.324357 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.324218 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:29.341879 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.341850 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz"] Oct 31 17:40:29.357924 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.357879 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:40:29.357924 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.357886 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:40:29.446721 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.446689 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e8ccc4b-swntz\" (UID: \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:29.446920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.446735 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzdkr\" (UniqueName: \"kubernetes.io/projected/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-kube-api-access-mzdkr\") pod \"maas-api-revoke-sub-3e8ccc4b-swntz\" (UID: \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:29.446920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.446767 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e8ccc4b-swntz\" (UID: \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:29.548205 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.548112 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e8ccc4b-swntz\" (UID: \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:29.548205 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.548167 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzdkr\" (UniqueName: \"kubernetes.io/projected/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-kube-api-access-mzdkr\") pod \"maas-api-revoke-sub-3e8ccc4b-swntz\" (UID: \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:29.548205 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.548200 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e8ccc4b-swntz\" (UID: \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:29.549010 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.548985 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e8ccc4b-swntz\" (UID: \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:29.550690 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.550607 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e8ccc4b-swntz\" (UID: \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:29.566064 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.566035 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzdkr\" (UniqueName: \"kubernetes.io/projected/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-kube-api-access-mzdkr\") pod \"maas-api-revoke-sub-3e8ccc4b-swntz\" (UID: \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:29.641381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.641347 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:29.950440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.950410 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz"] Oct 31 17:40:29.953677 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:40:29.953641 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d22b3c_64fb_4916_9448_a58a5fe8de9c.slice/crio-e0f4b9ce71704059acde024f956cf8c45461c86d6524219fdd2f2695d9bc08f6 WatchSource:0}: Error finding container e0f4b9ce71704059acde024f956cf8c45461c86d6524219fdd2f2695d9bc08f6: Status 404 returned error can't find the container with id e0f4b9ce71704059acde024f956cf8c45461c86d6524219fdd2f2695d9bc08f6 Oct 31 17:40:29.974598 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.974565 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" event={"ID":"e9d22b3c-64fb-4916-9448-a58a5fe8de9c","Type":"ContainerStarted","Data":"e0f4b9ce71704059acde024f956cf8c45461c86d6524219fdd2f2695d9bc08f6"} Oct 31 17:40:29.988443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:29.988418 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:40:30.985496 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:30.985458 7618 generic.go:356] "Generic (PLEG): container finished" podID="e9d22b3c-64fb-4916-9448-a58a5fe8de9c" containerID="52863280df042351b2b84ae0574e6890d67d0626f63c6375fe64639e32ac90e3" exitCode=0 Oct 31 17:40:30.985891 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:30.985537 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" event={"ID":"e9d22b3c-64fb-4916-9448-a58a5fe8de9c","Type":"ContainerDied","Data":"52863280df042351b2b84ae0574e6890d67d0626f63c6375fe64639e32ac90e3"} Oct 31 17:40:32.143817 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.143780 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq"] Oct 31 17:40:32.148704 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.148679 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.163934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.163904 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:32.179660 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.179620 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq"] Oct 31 17:40:32.287374 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.287291 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-xks-ca\") pod \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\" (UID: \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\") " Oct 31 17:40:32.287374 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.287356 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-kube-api-access-mzdkr\" (UniqueName: \"kubernetes.io/projected/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-kube-api-access-mzdkr\") pod \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\" (UID: \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\") " Oct 31 17:40:32.287639 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.287500 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-openshift-ca\") pod \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\" (UID: \"e9d22b3c-64fb-4916-9448-a58a5fe8de9c\") " Oct 31 17:40:32.287776 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.287752 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-envoy\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.287896 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.287787 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-data\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.287896 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.287815 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.288040 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.287904 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-openshift-ca" pod "e9d22b3c-64fb-4916-9448-a58a5fe8de9c" (UID: "e9d22b3c-64fb-4916-9448-a58a5fe8de9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:40:32.288040 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.287967 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx986\" (UniqueName: \"kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-kube-api-access-mx986\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.288147 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.288063 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istiod-ca-cert\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.288147 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.288104 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-credential-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.288147 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.288144 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-podinfo\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.288342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.288230 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-token\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.288342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.288292 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-certs\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.288451 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.288436 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:40:32.289647 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.289623 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-xks-ca" pod "e9d22b3c-64fb-4916-9448-a58a5fe8de9c" (UID: "e9d22b3c-64fb-4916-9448-a58a5fe8de9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:40:32.289737 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.289689 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-kube-api-access-mzdkr" pod "e9d22b3c-64fb-4916-9448-a58a5fe8de9c" (UID: "e9d22b3c-64fb-4916-9448-a58a5fe8de9c"). InnerVolumeSpecName "kube-api-access-mzdkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:40:32.390302 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.390247 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istiod-ca-cert\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.390491 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.390344 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-credential-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.390491 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.390378 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-podinfo\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.390491 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.390463 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-token\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.390678 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.390576 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-certs\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.390678 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.390655 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-envoy\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.393184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.390677 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-data\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.393184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.390712 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.393184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.390731 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-credential-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.393184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.390775 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx986\" (UniqueName: \"kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-kube-api-access-mx986\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.393184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.390984 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istiod-ca-cert\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.393184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.391139 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:40:32.393184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.391172 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzdkr\" (UniqueName: \"kubernetes.io/projected/e9d22b3c-64fb-4916-9448-a58a5fe8de9c-kube-api-access-mzdkr\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:40:32.393184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.391227 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-data\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.393184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.391334 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.393184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.391357 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-certs\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.393184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.393140 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-envoy\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.393932 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.393909 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-podinfo\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.408025 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.407994 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-token\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.408281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.408248 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx986\" (UniqueName: \"kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-kube-api-access-mx986\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.482007 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.481967 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:32.819707 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.819682 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq"] Oct 31 17:40:32.823498 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:40:32.823460 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e715d3f_f0cf_43ab_8fc0_b6413bb55451.slice/crio-00c26cce37d0cb13ec8fa4c236a6f34eac145329536dd68b01332e3b1ebdf492 WatchSource:0}: Error finding container 00c26cce37d0cb13ec8fa4c236a6f34eac145329536dd68b01332e3b1ebdf492: Status 404 returned error can't find the container with id 00c26cce37d0cb13ec8fa4c236a6f34eac145329536dd68b01332e3b1ebdf492 Oct 31 17:40:32.826543 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.826508 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:40:32.826646 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.826575 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:40:32.826646 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:32.826603 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:40:33.003867 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:33.003818 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" event={"ID":"e9d22b3c-64fb-4916-9448-a58a5fe8de9c","Type":"ContainerDied","Data":"e0f4b9ce71704059acde024f956cf8c45461c86d6524219fdd2f2695d9bc08f6"} Oct 31 17:40:33.004039 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:33.003926 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0f4b9ce71704059acde024f956cf8c45461c86d6524219fdd2f2695d9bc08f6" Oct 31 17:40:33.004039 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:33.004035 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz" Oct 31 17:40:33.008684 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:33.008645 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" event={"ID":"9e715d3f-f0cf-43ab-8fc0-b6413bb55451","Type":"ContainerStarted","Data":"162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44"} Oct 31 17:40:33.008830 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:33.008720 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" event={"ID":"9e715d3f-f0cf-43ab-8fc0-b6413bb55451","Type":"ContainerStarted","Data":"00c26cce37d0cb13ec8fa4c236a6f34eac145329536dd68b01332e3b1ebdf492"} Oct 31 17:40:33.060896 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:33.060829 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" podStartSLOduration=1.060807266 podStartE2EDuration="1.060807266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:40:33.055983963 +0000 UTC m=+754.803958430" watchObservedRunningTime="2026-10-31 17:40:33.060807266 +0000 UTC m=+754.808781733" Oct 31 17:40:33.482988 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:33.482947 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:33.484782 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:33.484749 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 17:40:33.484910 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:33.484813 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 17:40:34.482974 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:34.482937 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 17:40:34.483186 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:34.483012 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 17:40:35.483296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:35.483241 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 17:40:35.483786 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:35.483335 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 17:40:35.912231 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:35.912198 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:40:36.483496 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:36.483460 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 17:40:36.483900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:36.483514 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 17:40:36.557466 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:36.557374 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:36.570464 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:36.570431 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:37.075733 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:37.074196 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:40:37.493277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:37.482727 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 17:40:37.493277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:37.482808 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 17:40:38.488345 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:38.484810 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 17:40:38.488345 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:38.484877 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 17:40:38.721966 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:40:38.721928 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:40:39.485420 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:39.483539 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 17:40:39.485420 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:39.483609 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 17:40:40.603339 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:40.603304 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:40.603830 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:40.603669 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:40.604887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:40.604857 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:40:46.647844 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.647804 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t"] Oct 31 17:40:46.648607 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.648587 7618 cpu_manager.go:414] "containerMap: removing container" podUID="e9d22b3c-64fb-4916-9448-a58a5fe8de9c" containerName="revoke-keys" Oct 31 17:40:46.648707 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.648609 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d22b3c-64fb-4916-9448-a58a5fe8de9c" containerName="revoke-keys" Oct 31 17:40:46.648834 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.648820 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d22b3c-64fb-4916-9448-a58a5fe8de9c" containerName="revoke-keys" Oct 31 17:40:46.652513 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.652493 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.675685 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.675641 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t"] Oct 31 17:40:46.802851 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.802816 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288b5eba-33dc-4576-b70a-0e1b4da73004-istiod-ca-cert\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.803048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.802863 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-envoy\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.803048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.802960 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvqgh\" (UniqueName: \"kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-kube-api-access-vvqgh\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.803183 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.803060 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.803183 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.803136 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-data\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.803183 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.803166 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-certs\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.803359 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.803324 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-credential-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.803422 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.803409 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-podinfo\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.803476 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.803447 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-token\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.804309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.804292 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs"] Oct 31 17:40:46.818950 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.818921 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:46.830600 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.830573 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs"] Oct 31 17:40:46.905206 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905120 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmgvb\" (UniqueName: \"kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-kube-api-access-pmgvb\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:46.905206 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905168 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:46.905206 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905187 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-credential-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:46.905467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905337 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-podinfo\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.905467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905368 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-envoy\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:46.905467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905405 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-certs\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:46.905562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905481 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-token\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.905595 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905575 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288b5eba-33dc-4576-b70a-0e1b4da73004-istiod-ca-cert\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.905631 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905603 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-envoy\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.905631 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905623 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-token\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:46.905700 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905648 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvqgh\" (UniqueName: \"kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-kube-api-access-vvqgh\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.905888 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905867 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.905958 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905906 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istiod-ca-cert\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:46.906009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905956 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-data\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.906009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.905987 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-certs\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.906114 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.906081 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-podinfo\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:46.906194 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.906164 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-data\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:46.906277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.906218 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-credential-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.906277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.906218 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.906380 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.906347 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-data\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.906380 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.906361 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-certs\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.906490 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.906389 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288b5eba-33dc-4576-b70a-0e1b4da73004-istiod-ca-cert\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.906634 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.906615 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-credential-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.907871 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.907846 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-envoy\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.908002 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.907987 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-podinfo\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.923046 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.923017 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvqgh\" (UniqueName: \"kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-kube-api-access-vvqgh\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.923239 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.923216 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-token\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:46.971753 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:46.971720 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:47.007369 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.007332 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-token\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.007550 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.007430 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istiod-ca-cert\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.007550 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.007501 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-podinfo\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.007672 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.007559 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-data\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.007672 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.007623 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmgvb\" (UniqueName: \"kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-kube-api-access-pmgvb\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.007774 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.007672 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.007774 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.007697 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-credential-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.007774 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.007752 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-envoy\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.007914 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.007804 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-certs\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.008170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.008150 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-certs\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.008275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.008175 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-data\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.008332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.008297 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istiod-ca-cert\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.008565 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.008542 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-credential-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.008775 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.008757 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.010452 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.010429 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-envoy\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.010789 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.010769 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-podinfo\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.023950 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.023914 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-token\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.025016 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.024933 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmgvb\" (UniqueName: \"kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-kube-api-access-pmgvb\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.136704 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.136669 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:47.301594 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.301459 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t"] Oct 31 17:40:47.304881 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:40:47.304853 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288b5eba_33dc_4576_b70a_0e1b4da73004.slice/crio-57d6b3032ae35388a4b7dadb4a2c989e8efcff9a472c54eba75029de884b9a66 WatchSource:0}: Error finding container 57d6b3032ae35388a4b7dadb4a2c989e8efcff9a472c54eba75029de884b9a66: Status 404 returned error can't find the container with id 57d6b3032ae35388a4b7dadb4a2c989e8efcff9a472c54eba75029de884b9a66 Oct 31 17:40:47.307620 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.307584 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:40:47.307725 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.307654 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:40:47.307725 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.307683 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:40:47.453278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.452649 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs"] Oct 31 17:40:47.460587 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.460543 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:40:47.460711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.460632 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:40:47.460711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:47.460672 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:40:48.183788 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:48.183740 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" event={"ID":"8ccce903-d0aa-4873-adcc-ab8d4d1df86b","Type":"ContainerStarted","Data":"a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3"} Oct 31 17:40:48.184333 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:48.183816 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" event={"ID":"8ccce903-d0aa-4873-adcc-ab8d4d1df86b","Type":"ContainerStarted","Data":"5d09289ac3e6d2b107298679125b55e973cfade4e908d774eda0aaf8f1fdff0c"} Oct 31 17:40:48.185866 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:48.185833 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" event={"ID":"288b5eba-33dc-4576-b70a-0e1b4da73004","Type":"ContainerStarted","Data":"d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243"} Oct 31 17:40:48.185986 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:48.185879 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" event={"ID":"288b5eba-33dc-4576-b70a-0e1b4da73004","Type":"ContainerStarted","Data":"57d6b3032ae35388a4b7dadb4a2c989e8efcff9a472c54eba75029de884b9a66"} Oct 31 17:40:48.238274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:48.237568 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" podStartSLOduration=2.237548944 podStartE2EDuration="2.237548944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:40:48.236771196 +0000 UTC m=+769.984745665" watchObservedRunningTime="2026-10-31 17:40:48.237548944 +0000 UTC m=+769.985523412" Oct 31 17:40:48.281659 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:48.281383 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" podStartSLOduration=2.281359256 podStartE2EDuration="2.281359256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:40:48.277793553 +0000 UTC m=+770.025768021" watchObservedRunningTime="2026-10-31 17:40:48.281359256 +0000 UTC m=+770.029333735" Oct 31 17:40:48.972277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:48.972150 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:48.973861 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:48.973828 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 17:40:48.973967 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:48.973888 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" podUID="288b5eba-33dc-4576-b70a-0e1b4da73004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 17:40:49.146278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:49.142642 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:49.161332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:49.160449 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 17:40:49.161332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:49.160554 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" podUID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 17:40:49.974277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:49.973644 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 17:40:49.974277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:49.973707 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" podUID="288b5eba-33dc-4576-b70a-0e1b4da73004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 17:40:50.137694 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:50.137654 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 17:40:50.137991 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:50.137960 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" podUID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 17:40:50.974276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:50.973687 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 17:40:50.974276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:50.973758 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" podUID="288b5eba-33dc-4576-b70a-0e1b4da73004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 17:40:51.150571 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:51.143776 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 17:40:51.150571 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:51.143843 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" podUID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 17:40:51.974278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:51.973683 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 17:40:51.974278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:51.973753 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" podUID="288b5eba-33dc-4576-b70a-0e1b4da73004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 17:40:52.137279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:52.136983 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 17:40:52.137279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:52.137045 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" podUID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 17:40:52.974018 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:52.972937 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 17:40:52.974018 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:52.972999 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" podUID="288b5eba-33dc-4576-b70a-0e1b4da73004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 17:40:53.140277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:53.139659 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 17:40:53.140277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:53.139728 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" podUID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 17:40:53.972647 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:53.972609 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 17:40:53.972837 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:53.972680 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" podUID="288b5eba-33dc-4576-b70a-0e1b4da73004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 17:40:54.137762 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:54.137717 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 17:40:54.137967 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:54.137788 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" podUID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 17:40:54.978092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:54.978055 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:54.978603 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:54.978460 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:54.979242 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:54.979224 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:40:55.143272 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.142944 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:55.143414 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.143377 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:55.146290 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.144473 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:40:55.453931 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.453881 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz"] Oct 31 17:40:55.460440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.460416 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.473613 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.473539 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cf0a86\"/\"distinct-2-w3-cf0a86-kserve-self-signed-certs\"" Oct 31 17:40:55.476136 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.476108 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz"] Oct 31 17:40:55.518729 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.518689 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sbfz\" (UniqueName: \"kubernetes.io/projected/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-kube-api-access-7sbfz\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.518942 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.518754 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-model-cache\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.518942 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.518903 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tmp-dir\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.519057 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.518988 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-dshm\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.519112 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.519051 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tls-certs\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.519169 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.519146 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-home\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.620459 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.620425 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tmp-dir\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.620652 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.620581 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-dshm\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.620652 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.620644 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tls-certs\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.620763 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.620745 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-home\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.620872 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.620853 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sbfz\" (UniqueName: \"kubernetes.io/projected/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-kube-api-access-7sbfz\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.620940 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.620902 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-model-cache\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.621079 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.620902 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tmp-dir\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.621527 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.621500 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-home\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.621652 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.621541 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-model-cache\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.624046 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.624020 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-dshm\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.624432 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.624409 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tls-certs\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.638795 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.638734 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sbfz\" (UniqueName: \"kubernetes.io/projected/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-kube-api-access-7sbfz\") pod \"distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:55.780091 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:55.777980 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:40:56.119438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:56.119407 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz"] Oct 31 17:40:56.121176 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:40:56.121148 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05ee47cc_590f_4de3_ab6d_0ce5d97c41d5.slice/crio-270e8553ec8fe25c5f3c053486c223ddb7830fe79e94bc5195373ac55497ff34 WatchSource:0}: Error finding container 270e8553ec8fe25c5f3c053486c223ddb7830fe79e94bc5195373ac55497ff34: Status 404 returned error can't find the container with id 270e8553ec8fe25c5f3c053486c223ddb7830fe79e94bc5195373ac55497ff34 Oct 31 17:40:56.287704 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:56.287664 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" event={"ID":"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5","Type":"ContainerStarted","Data":"03469e47bea4bbcd68ca439038b9808122d4ef0ac93e1634cc0f931828a80565"} Oct 31 17:40:56.287878 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:56.287720 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" event={"ID":"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5","Type":"ContainerStarted","Data":"270e8553ec8fe25c5f3c053486c223ddb7830fe79e94bc5195373ac55497ff34"} Oct 31 17:40:56.323491 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:56.323437 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podStartSLOduration=1.323422932 podStartE2EDuration="1.323422932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:40:56.322172856 +0000 UTC m=+778.070147327" watchObservedRunningTime="2026-10-31 17:40:56.323422932 +0000 UTC m=+778.071397398" Oct 31 17:40:56.778727 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:40:56.778693 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:41:02.137563 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.137518 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2"] Oct 31 17:41:02.262277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.261663 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.477278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.468236 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-envoy\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.477278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.468320 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-podinfo\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.477278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.468491 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-data\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.477278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.468548 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-credential-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.477278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.468593 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-certs\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.477278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.468633 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-token\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.477278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.468659 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istiod-ca-cert\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.477278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.468764 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjn6m\" (UniqueName: \"kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-kube-api-access-wjn6m\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.477278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.468796 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.527020 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.526976 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7"] Oct 31 17:41:02.583381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.583340 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjn6m\" (UniqueName: \"kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-kube-api-access-wjn6m\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.583691 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.583653 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.584009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.583992 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-envoy\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.584243 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.584222 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-podinfo\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.584544 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.584522 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-data\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.584710 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.584692 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-credential-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.584893 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.584863 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-certs\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.585839 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.585821 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-token\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.586001 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.585982 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istiod-ca-cert\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.586314 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.585661 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-credential-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.586455 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.585749 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-data\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.586577 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.585372 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-certs\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.587016 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.584574 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.589085 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.589065 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istiod-ca-cert\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.647313 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.603794 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.647313 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.625978 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjn6m\" (UniqueName: \"kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-kube-api-access-wjn6m\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.663273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.648095 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-podinfo\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.663273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.663220 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-token\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.663906 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.663828 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2"] Oct 31 17:41:02.725277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.717994 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-certs\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.725576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.718658 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-envoy\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.725767 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.725590 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b39ed1-7e86-42d4-9241-94a7e369638b-istiod-ca-cert\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.725939 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.725904 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-envoy\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.726119 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.726100 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtd7w\" (UniqueName: \"kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-kube-api-access-jtd7w\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.730400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.730337 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-credential-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.730569 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.730549 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-data\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.730744 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.730725 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.730971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.730950 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-podinfo\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.731212 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.731191 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-token\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.760327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.760297 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7"] Oct 31 17:41:02.783537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.773503 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn"] Oct 31 17:41:02.783537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.778715 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:02.850378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.849200 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-token\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.850378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.849341 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-certs\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.850378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.849434 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b39ed1-7e86-42d4-9241-94a7e369638b-istiod-ca-cert\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.850378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.849467 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-envoy\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.850378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.849497 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtd7w\" (UniqueName: \"kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-kube-api-access-jtd7w\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.850378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.849559 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-credential-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.850378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.849592 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-data\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.850378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.849680 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.850378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.849740 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-podinfo\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.862342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.851790 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b39ed1-7e86-42d4-9241-94a7e369638b-istiod-ca-cert\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.862342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.853058 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-certs\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.862342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.853322 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-credential-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.862342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.854498 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn"] Oct 31 17:41:02.862342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.854660 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:02.862342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.856435 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-data\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.862342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.856820 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-socket\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.889562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.882708 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-podinfo\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.907646 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.907607 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-envoy\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.950535 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.927854 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-token\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.960332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.952974 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtd7w\" (UniqueName: \"kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-kube-api-access-jtd7w\") pod \"e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.960558 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.958101 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92160482-537c-4d9d-b55c-e8200f78f40a-istio-podinfo\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:02.973719 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.973506 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:02.976271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.975425 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92160482-537c-4d9d-b55c-e8200f78f40a-istiod-ca-cert\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:02.976271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.975490 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-credential-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:02.976271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.975537 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-certs\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:02.976271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.975569 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-istio-token\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:02.976271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.975650 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-envoy\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:02.976271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.975766 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-data\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:02.976271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.975836 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6lpx\" (UniqueName: \"kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-kube-api-access-j6lpx\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:02.976271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:02.975944 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.078460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.077131 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92160482-537c-4d9d-b55c-e8200f78f40a-istio-podinfo\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.078460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.077214 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92160482-537c-4d9d-b55c-e8200f78f40a-istiod-ca-cert\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.078460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.077246 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-credential-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.078460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.077303 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-certs\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.078460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.077335 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-istio-token\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.078460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.077407 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-envoy\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.078460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.077513 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-data\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.078460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.077598 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lpx\" (UniqueName: \"kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-kube-api-access-j6lpx\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.078460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.077744 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.078460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.078294 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.093287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.090682 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-credential-socket\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.093287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.091539 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92160482-537c-4d9d-b55c-e8200f78f40a-istiod-ca-cert\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.093287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.092219 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-data\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.094036 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.093718 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-certs\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.117398 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.117343 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-envoy\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.178333 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.120855 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92160482-537c-4d9d-b55c-e8200f78f40a-istio-podinfo\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.222926 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.222886 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-istio-token\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.261697 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.261589 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lpx\" (UniqueName: \"kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-kube-api-access-j6lpx\") pod \"e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.329423 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.327387 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:03.369756 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.361379 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:03.381388 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.381316 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:03.619334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.599148 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:03.619334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.618547 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:03.619551 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.619354 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Oct 31 17:41:03.671788 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.671363 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:03.776292 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.762361 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:03.795105 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.794367 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:03.795105 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:03.794778 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 17:41:04.034323 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.019706 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2"] Oct 31 17:41:04.087820 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.080076 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:04.087820 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.080174 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:04.087820 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.080231 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:04.607607 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.607574 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7"] Oct 31 17:41:04.629770 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.629737 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:04.629879 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.629818 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:04.629879 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.629862 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:04.661165 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.654628 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn"] Oct 31 17:41:04.669358 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:41:04.669321 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92160482_537c_4d9d_b55c_e8200f78f40a.slice/crio-e49c0fb171d3ef615d1a7374ec0195f07b97fff05067d38ccad3d992083578b3 WatchSource:0}: Error finding container e49c0fb171d3ef615d1a7374ec0195f07b97fff05067d38ccad3d992083578b3: Status 404 returned error can't find the container with id e49c0fb171d3ef615d1a7374ec0195f07b97fff05067d38ccad3d992083578b3 Oct 31 17:41:04.671625 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.671592 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:04.671738 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.671714 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:04.671796 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.671759 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:04.717293 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.717151 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" event={"ID":"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2","Type":"ContainerStarted","Data":"5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79"} Oct 31 17:41:04.717293 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.717218 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" event={"ID":"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2","Type":"ContainerStarted","Data":"10e4fd50ae5f2770a9edfb7dfd793933d3891355b89b31ef4e97d2468b251768"} Oct 31 17:41:04.719561 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.719495 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" event={"ID":"92160482-537c-4d9d-b55c-e8200f78f40a","Type":"ContainerStarted","Data":"e49c0fb171d3ef615d1a7374ec0195f07b97fff05067d38ccad3d992083578b3"} Oct 31 17:41:04.721026 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.721001 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" event={"ID":"61b39ed1-7e86-42d4-9241-94a7e369638b","Type":"ContainerStarted","Data":"ba35dedaccb5de2c92188be42d0833fe30601c220b372de8c226f6e24eb6ecb3"} Oct 31 17:41:04.760658 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.759873 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" podStartSLOduration=2.759854426 podStartE2EDuration="2.759854426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:04.754997989 +0000 UTC m=+786.502972455" watchObservedRunningTime="2026-10-31 17:41:04.759854426 +0000 UTC m=+786.507828894" Oct 31 17:41:04.779436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:04.779335 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:05.737414 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:05.737372 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" event={"ID":"61b39ed1-7e86-42d4-9241-94a7e369638b","Type":"ContainerStarted","Data":"b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66"} Oct 31 17:41:05.742350 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:05.742304 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" event={"ID":"92160482-537c-4d9d-b55c-e8200f78f40a","Type":"ContainerStarted","Data":"4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885"} Oct 31 17:41:05.792399 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:05.792366 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:41:05.797134 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:05.796843 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 17:41:05.797134 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:05.796903 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 17:41:05.811075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:05.811011 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" podStartSLOduration=3.810990674 podStartE2EDuration="3.810990674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:05.794586092 +0000 UTC m=+787.542560575" watchObservedRunningTime="2026-10-31 17:41:05.810990674 +0000 UTC m=+787.558965144" Oct 31 17:41:05.843966 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:05.843899 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" podStartSLOduration=3.84387707 podStartE2EDuration="3.84387707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:05.842669955 +0000 UTC m=+787.590644426" watchObservedRunningTime="2026-10-31 17:41:05.84387707 +0000 UTC m=+787.591851540" Oct 31 17:41:05.888814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:05.888779 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:41:05.978858 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:05.977791 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 17:41:05.978858 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:05.977858 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 17:41:05.978858 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:05.978123 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:06.388111 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:06.388076 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 17:41:06.388516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:06.388453 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 17:41:06.415523 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:06.414017 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:06.781608 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:06.781572 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 17:41:06.782335 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:06.782305 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 17:41:06.979661 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:06.979621 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 17:41:06.979931 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:06.979904 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 17:41:07.385271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:07.384320 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 17:41:07.385271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:07.384879 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 17:41:07.513428 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:07.513369 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:07.763453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:07.763410 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:07.812126 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:07.791662 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 17:41:07.813184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:07.813139 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 17:41:07.831535 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:07.831479 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:07.972426 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:07.972380 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": context deadline exceeded" Oct 31 17:41:07.979344 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:07.979309 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 17:41:07.979633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:07.979606 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 17:41:07.981372 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:07.981335 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:08.300574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:08.300528 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:08.393274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:08.392507 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 17:41:08.393274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:08.392569 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 17:41:08.503276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:08.502573 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:41:08.859289 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:08.858635 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 17:41:08.859289 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:08.858702 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 17:41:08.984275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:08.976812 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 17:41:08.984275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:08.976872 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 17:41:09.113343 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:41:09.110887 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:41:09.386335 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:09.383048 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 17:41:09.393768 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:09.393725 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 17:41:09.784274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:09.783571 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 17:41:09.784274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:09.783632 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 17:41:10.044621 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.037687 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 17:41:10.044621 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.037753 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 17:41:10.261278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.252869 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp"] Oct 31 17:41:10.269626 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.268512 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:10.288400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.286278 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:41:10.349278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.307677 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:41:10.349278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.335739 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp"] Oct 31 17:41:10.349278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.343503 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkh9q\" (UniqueName: \"kubernetes.io/projected/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-kube-api-access-dkh9q\") pod \"maas-api-revoke-sub-200c6dfe-8v5qp\" (UID: \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:10.349278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.343579 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-200c6dfe-8v5qp\" (UID: \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:10.349278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.343680 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-200c6dfe-8v5qp\" (UID: \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:10.512005 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.511943 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 17:41:10.512345 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.512312 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 17:41:10.609493 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.608164 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkh9q\" (UniqueName: \"kubernetes.io/projected/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-kube-api-access-dkh9q\") pod \"maas-api-revoke-sub-200c6dfe-8v5qp\" (UID: \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:10.609493 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.608304 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-200c6dfe-8v5qp\" (UID: \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:10.609493 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.608440 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-200c6dfe-8v5qp\" (UID: \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:10.680291 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.679746 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-200c6dfe-8v5qp\" (UID: \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:10.700926 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.700883 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkh9q\" (UniqueName: \"kubernetes.io/projected/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-kube-api-access-dkh9q\") pod \"maas-api-revoke-sub-200c6dfe-8v5qp\" (UID: \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:10.725283 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.717065 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-200c6dfe-8v5qp\" (UID: \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:10.782746 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.782702 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 17:41:10.783079 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.783031 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 17:41:10.975534 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.975496 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 17:41:10.975833 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:10.975802 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 17:41:11.016827 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:11.016503 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:11.394336 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:11.390143 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 17:41:11.394336 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:11.390230 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 17:41:11.779717 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:11.779554 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 17:41:11.780044 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:11.779632 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 17:41:11.902223 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:11.901903 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp"] Oct 31 17:41:11.920641 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:41:11.905668 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1ee1dd_f558_4cfe_a2f9_81ebc3e63291.slice/crio-711d270c4d9d38466e49a49aae76147ef249f2bae1432d79284fbebdab99eb36 WatchSource:0}: Error finding container 711d270c4d9d38466e49a49aae76147ef249f2bae1432d79284fbebdab99eb36: Status 404 returned error can't find the container with id 711d270c4d9d38466e49a49aae76147ef249f2bae1432d79284fbebdab99eb36 Oct 31 17:41:11.975790 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:11.975653 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 17:41:11.975790 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:11.975758 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 17:41:12.303969 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:12.303932 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" event={"ID":"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291","Type":"ContainerStarted","Data":"711d270c4d9d38466e49a49aae76147ef249f2bae1432d79284fbebdab99eb36"} Oct 31 17:41:12.384178 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:12.384066 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 17:41:12.384178 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:12.384139 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 17:41:12.987277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:12.975297 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 17:41:12.987277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:12.975364 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 17:41:13.114263 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:13.114227 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:13.115581 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:13.115554 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:13.118143 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:13.118093 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:41:13.346618 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:13.346520 7618 generic.go:356] "Generic (PLEG): container finished" podID="dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291" containerID="20637b2690dd6c202d415338d2ca53c03e1e7dbb8d799c59bd15f394e1cede8a" exitCode=0 Oct 31 17:41:13.346868 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:13.346660 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" event={"ID":"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291","Type":"ContainerDied","Data":"20637b2690dd6c202d415338d2ca53c03e1e7dbb8d799c59bd15f394e1cede8a"} Oct 31 17:41:13.466342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:13.466313 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:13.467246 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:13.467209 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:13.468892 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:13.468874 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:41:13.980242 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:13.980213 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:13.980626 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:13.980602 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:13.981493 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:13.981470 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:41:14.528850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:14.528818 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:14.584074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:14.584037 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-openshift-ca\") pod \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\" (UID: \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\") " Oct 31 17:41:14.584415 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:14.584379 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-openshift-ca" pod "dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291" (UID: "dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:41:14.584415 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:14.584403 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-kube-api-access-dkh9q\" (UniqueName: \"kubernetes.io/projected/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-kube-api-access-dkh9q\") pod \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\" (UID: \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\") " Oct 31 17:41:14.584600 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:14.584456 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-xks-ca\") pod \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\" (UID: \"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291\") " Oct 31 17:41:14.585289 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:14.585234 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:41:14.588103 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:14.587865 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-xks-ca" pod "dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291" (UID: "dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:41:14.591263 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:14.591217 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-kube-api-access-dkh9q" pod "dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291" (UID: "dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291"). InnerVolumeSpecName "kube-api-access-dkh9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:41:14.686530 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:14.686496 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkh9q\" (UniqueName: \"kubernetes.io/projected/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-kube-api-access-dkh9q\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:41:14.686530 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:14.686525 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:41:15.365907 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:15.365871 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" Oct 31 17:41:15.365907 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:15.365888 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp" event={"ID":"dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291","Type":"ContainerDied","Data":"711d270c4d9d38466e49a49aae76147ef249f2bae1432d79284fbebdab99eb36"} Oct 31 17:41:15.366198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:15.365964 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="711d270c4d9d38466e49a49aae76147ef249f2bae1432d79284fbebdab99eb36" Oct 31 17:41:16.628298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.626643 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb"] Oct 31 17:41:16.628298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.627678 7618 cpu_manager.go:414] "containerMap: removing container" podUID="dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291" containerName="revoke-keys" Oct 31 17:41:16.628298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.627697 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291" containerName="revoke-keys" Oct 31 17:41:16.628298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.627792 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291" containerName="revoke-keys" Oct 31 17:41:16.631647 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.631623 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.650130 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.650101 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b78c31-openshift-default-dockercfg-7pl9w\"" Oct 31 17:41:16.652621 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.652573 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb"] Oct 31 17:41:16.715710 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.715659 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-envoy\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.715710 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.715711 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-data\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.715963 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.715822 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgq2l\" (UniqueName: \"kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-kube-api-access-dgq2l\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.715963 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.715861 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istiod-ca-cert\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.716077 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.716000 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-certs\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.716130 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.716069 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-token\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.716182 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.716163 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-credential-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.716235 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.716195 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-podinfo\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.716400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.716378 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.818224 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.818190 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-credential-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.818224 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.818234 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-podinfo\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.818528 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.818355 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.818528 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.818451 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-envoy\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.818528 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.818478 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-data\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.818689 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.818532 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgq2l\" (UniqueName: \"kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-kube-api-access-dgq2l\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.818689 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.818571 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istiod-ca-cert\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.818689 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.818661 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-certs\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.818846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.818692 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-token\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.818898 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.818868 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.818944 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.818897 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-data\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.819352 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.819329 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-certs\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.819656 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.819629 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istiod-ca-cert\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.819761 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.819716 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-credential-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.821332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.821308 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-envoy\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.822032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.822010 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-podinfo\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.836209 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.836174 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-token\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.836639 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.836611 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgq2l\" (UniqueName: \"kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-kube-api-access-dgq2l\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:16.912668 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.912628 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p"] Oct 31 17:41:16.924045 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.919408 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:16.939626 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.937393 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p"] Oct 31 17:41:16.949067 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:16.949027 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:17.012628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.012589 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5"] Oct 31 17:41:17.019354 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.017632 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.021154 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.021121 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d56tp\" (UniqueName: \"kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-kube-api-access-d56tp\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.021302 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.021268 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-token\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.021374 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.021316 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-credential-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.022040 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.021743 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istiod-ca-cert\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.022040 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.021996 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.022208 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.022082 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-envoy\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.022208 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.022126 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-certs\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.022208 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.022151 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-podinfo\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.022414 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.022211 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-data\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.057537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.057336 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5"] Oct 31 17:41:17.124118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124034 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-podinfo\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.124338 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124146 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-token\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.124338 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124174 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-token\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.124338 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124205 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-credential-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.124338 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124223 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istiod-ca-cert\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.124338 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124240 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-envoy\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.124338 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124280 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58cl8\" (UniqueName: \"kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-kube-api-access-58cl8\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.124680 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124357 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.124680 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124420 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.124680 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124491 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-envoy\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.124680 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124521 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-certs\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.124680 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124546 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-podinfo\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.124680 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124582 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-certs\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.124680 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124631 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-data\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.125002 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124748 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istiod-ca-cert\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.125002 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124806 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-credential-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.125002 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124883 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-data\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.125002 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.124929 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d56tp\" (UniqueName: \"kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-kube-api-access-d56tp\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.125871 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.125846 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-credential-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.126448 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.126376 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istiod-ca-cert\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.126711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.126688 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-socket\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.127320 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.127298 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-data\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.127739 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.127720 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-certs\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.133298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.133057 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-podinfo\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.135739 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.135714 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-envoy\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.143050 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.143005 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-token\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.149534 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.149488 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d56tp\" (UniqueName: \"kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-kube-api-access-d56tp\") pod \"e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.226208 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.226114 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-token\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.226208 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.226168 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-envoy\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.226208 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.226197 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58cl8\" (UniqueName: \"kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-kube-api-access-58cl8\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.226522 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.226276 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.226522 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.226387 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-certs\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.226522 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.226446 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istiod-ca-cert\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.226522 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.226481 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-credential-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.226729 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.226538 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-data\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.226729 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.226647 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-podinfo\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.226836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.226797 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.227309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.227157 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-credential-socket\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.227309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.227177 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-certs\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.227309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.227210 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istiod-ca-cert\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.227309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.227247 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-data\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.228634 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.228606 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-envoy\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.229066 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.229046 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-podinfo\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.242894 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.242848 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:17.243195 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.243174 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-token\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.243312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.243288 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58cl8\" (UniqueName: \"kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-kube-api-access-58cl8\") pod \"e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.340732 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:41:17.340694 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625c35a6_ff1d_4d4a_aa3c_cd126f7e6dcd.slice/crio-4321fb08d870e3d548d7bf83a4c09862e863d7569d37dfca13141dd25ba2f8a7 WatchSource:0}: Error finding container 4321fb08d870e3d548d7bf83a4c09862e863d7569d37dfca13141dd25ba2f8a7: Status 404 returned error can't find the container with id 4321fb08d870e3d548d7bf83a4c09862e863d7569d37dfca13141dd25ba2f8a7 Oct 31 17:41:17.344226 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.343841 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb"] Oct 31 17:41:17.346725 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.346690 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:17.346943 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.346922 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:17.347069 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.347054 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:17.367120 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.366585 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:17.394319 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.393375 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" event={"ID":"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd","Type":"ContainerStarted","Data":"4321fb08d870e3d548d7bf83a4c09862e863d7569d37dfca13141dd25ba2f8a7"} Oct 31 17:41:17.556886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.556847 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p"] Oct 31 17:41:17.557374 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:41:17.557333 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea2ccc1_eee0_4921_9146_78fcd7790bdf.slice/crio-63bb8eda7b0037b32b8b15823bf7f3644ce28e727f5e558f446180183534322b WatchSource:0}: Error finding container 63bb8eda7b0037b32b8b15823bf7f3644ce28e727f5e558f446180183534322b: Status 404 returned error can't find the container with id 63bb8eda7b0037b32b8b15823bf7f3644ce28e727f5e558f446180183534322b Oct 31 17:41:17.563351 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.562466 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:17.563351 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.562545 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:17.563351 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.562586 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:17.703246 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.703207 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5"] Oct 31 17:41:17.707978 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:41:17.707947 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7b20a8_644d_45fb_8055_4f0a8a90c48d.slice/crio-01314b38020e463d9e1cd718bf5949ccb69140d5b67fb699b9afe0b4faee0fb3 WatchSource:0}: Error finding container 01314b38020e463d9e1cd718bf5949ccb69140d5b67fb699b9afe0b4faee0fb3: Status 404 returned error can't find the container with id 01314b38020e463d9e1cd718bf5949ccb69140d5b67fb699b9afe0b4faee0fb3 Oct 31 17:41:17.711287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.711215 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:17.711389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.711314 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:17.711389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:17.711364 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:18.409305 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:18.409116 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" event={"ID":"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd","Type":"ContainerStarted","Data":"515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8"} Oct 31 17:41:18.425278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:18.414653 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" event={"ID":"4b7b20a8-644d-45fb-8055-4f0a8a90c48d","Type":"ContainerStarted","Data":"8387da5b989ca9d565255372597055dbe84b41509cf4ad5935c7898570e69b8e"} Oct 31 17:41:18.425278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:18.414726 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" event={"ID":"4b7b20a8-644d-45fb-8055-4f0a8a90c48d","Type":"ContainerStarted","Data":"01314b38020e463d9e1cd718bf5949ccb69140d5b67fb699b9afe0b4faee0fb3"} Oct 31 17:41:18.435288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:18.434376 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" event={"ID":"9ea2ccc1-eee0-4921-9146-78fcd7790bdf","Type":"ContainerStarted","Data":"c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e"} Oct 31 17:41:18.435288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:18.434445 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" event={"ID":"9ea2ccc1-eee0-4921-9146-78fcd7790bdf","Type":"ContainerStarted","Data":"63bb8eda7b0037b32b8b15823bf7f3644ce28e727f5e558f446180183534322b"} Oct 31 17:41:18.503125 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:18.503061 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podStartSLOduration=2.503045823 podStartE2EDuration="2.503045823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:18.499937018 +0000 UTC m=+800.247911485" watchObservedRunningTime="2026-10-31 17:41:18.503045823 +0000 UTC m=+800.251020361" Oct 31 17:41:18.504092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:18.504062 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podStartSLOduration=2.504052089 podStartE2EDuration="2.504052089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:18.459599301 +0000 UTC m=+800.207573769" watchObservedRunningTime="2026-10-31 17:41:18.504052089 +0000 UTC m=+800.252026559" Oct 31 17:41:18.582273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:18.581291 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podStartSLOduration=2.581267707 podStartE2EDuration="2.581267707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:18.559723304 +0000 UTC m=+800.307697774" watchObservedRunningTime="2026-10-31 17:41:18.581267707 +0000 UTC m=+800.329242173" Oct 31 17:41:18.983289 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:18.970991 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:18.983289 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:18.983215 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 17:41:18.983839 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:18.983292 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 17:41:19.274355 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:19.254471 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:19.274355 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:19.255335 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 17:41:19.274355 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:19.255391 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 17:41:19.370300 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:19.368587 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:19.370300 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:19.368691 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 17:41:19.370300 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:19.368734 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 17:41:19.777288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:19.774357 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:19.853290 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:19.846339 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:19.952276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:19.951680 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 17:41:19.952276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:19.951748 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 17:41:19.978276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:19.977677 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 17:41:20.252170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.251716 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 17:41:20.252170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.251782 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 17:41:20.413294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.367647 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:20.414143 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.413637 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:20.480393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.416116 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 17:41:20.480393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.416186 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 17:41:20.505513 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.505026 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:20.588276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.571273 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 17:41:20.790353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.788209 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:20.790353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.788913 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:41:20.875288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.874173 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:20.875288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.874994 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:41:21.010144 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.996377 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 17:41:21.010144 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.996445 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 17:41:21.010144 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:20.997374 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:41:21.094883 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.094577 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:21.215109 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.215076 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:41:21.293853 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.293395 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 17:41:21.293853 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.293452 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 17:41:21.535485 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.535426 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:21.603278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.585366 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:21.603278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.592012 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:21.603278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.597102 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:41:21.603278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.597179 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:41:21.655309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.643472 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 17:41:21.655309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.643541 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 17:41:21.801550 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.791752 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:21.895376 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.895234 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:41:21.973132 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.973084 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 17:41:21.980440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:21.980400 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 17:41:22.201340 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.200746 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:41:22.257278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.244659 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 17:41:22.257278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.244723 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 17:41:22.344582 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.344541 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:22.378413 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.377564 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 17:41:22.378413 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.377629 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 17:41:22.659576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.659535 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd"] Oct 31 17:41:22.751276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.737808 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:22.804577 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.804545 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:41:22.805032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.805011 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:41:22.805802 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.805771 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:22.807158 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.807129 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:22.833191 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.831695 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:22.833191 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.832529 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:22.859337 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.859208 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-644484c9-m57xd\" (UID: \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:22.859555 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.859537 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-644484c9-m57xd\" (UID: \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:22.859644 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.859586 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9c74\" (UniqueName: \"kubernetes.io/projected/495ff2b5-327a-424e-a596-9c2b3dbfaf74-kube-api-access-s9c74\") pod \"maas-api-revoke-sub-644484c9-m57xd\" (UID: \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:22.915758 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.915352 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd"] Oct 31 17:41:22.915758 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.915481 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:41:22.967596 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.967558 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-644484c9-m57xd\" (UID: \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:22.987432 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.987397 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9c74\" (UniqueName: \"kubernetes.io/projected/495ff2b5-327a-424e-a596-9c2b3dbfaf74-kube-api-access-s9c74\") pod \"maas-api-revoke-sub-644484c9-m57xd\" (UID: \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:22.988121 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.988096 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-644484c9-m57xd\" (UID: \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:23.017287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.978951 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:23.021276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.967900 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 17:41:23.040684 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.040637 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 17:41:23.040969 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.978900 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-644484c9-m57xd\" (UID: \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:23.041096 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.011326 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:41:23.041324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.013900 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 17:41:23.041568 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:22.988970 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-644484c9-m57xd\" (UID: \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:23.122533 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.107407 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9c74\" (UniqueName: \"kubernetes.io/projected/495ff2b5-327a-424e-a596-9c2b3dbfaf74-kube-api-access-s9c74\") pod \"maas-api-revoke-sub-644484c9-m57xd\" (UID: \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:23.183327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.182321 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:23.247808 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.247347 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 17:41:23.247808 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.247404 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 17:41:23.383275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.380209 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 17:41:23.383275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.380286 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 17:41:23.383275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.380845 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:23.383275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.381702 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:41:23.485789 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.473358 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:23.485789 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.474505 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:41:23.518276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.516356 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:23.836595 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.834361 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:23.836595 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.835059 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:41:23.935644 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:23.933423 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:41:24.074702 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.074665 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 17:41:24.211641 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.211598 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 17:41:24.465588 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.445414 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 17:41:24.465588 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.445492 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 17:41:24.492286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.476491 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 17:41:24.492286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.476556 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 17:41:24.531286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.524683 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 17:41:24.843927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.843816 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:24.962276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.952000 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:41:24.962276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.952063 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd"] Oct 31 17:41:24.963006 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.962961 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:24.978439 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.978386 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:24.981087 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:24.981054 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 17:41:25.045510 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.045459 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 17:41:25.045815 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.005381 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:25.049161 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.000000 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 31 17:41:25.072930 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.072872 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:25.234816 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.234769 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 17:41:25.262280 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.237325 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:25.277316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.272502 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 17:41:25.277316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.272563 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 17:41:25.438134 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.438087 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:25.491747 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.439708 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" event={"ID":"495ff2b5-327a-424e-a596-9c2b3dbfaf74","Type":"ContainerStarted","Data":"09b5462368f323089575c9bac1c058223b65707132d258d23088bb553c3f00f9"} Oct 31 17:41:25.491747 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.451848 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 17:41:25.492273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.491777 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 17:41:25.492273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.477330 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:25.505471 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.505345 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qsm6h" podUID="60303a6b-433f-4b1c-8e2e-19e901fbd933" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:25.525403 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.525351 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 17:41:25.761413 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.760446 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:25.851506 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.851449 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:25.887325 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.869344 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 17:41:25.958276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.955635 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 17:41:25.958276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.955701 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 17:41:25.979401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.979353 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:25.980190 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.980166 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:41:25.982988 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.982329 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:25.982988 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:25.982597 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:41:26.081381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.055786 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 17:41:26.309285 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.093630 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 17:41:26.309285 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.307921 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:41:26.309285 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.307990 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:41:26.390638 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.107421 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:26.426875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.281342 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:26.427378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.248122 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 17:41:26.459315 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.459271 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 17:41:26.479854 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.248168 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:26.480368 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.374372 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 17:41:26.480569 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.480541 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 17:41:26.494385 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.494343 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:26.495276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.495229 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 17:41:26.497365 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.497340 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:41:26.497851 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.497834 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:41:26.518517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.518475 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:41:26.724467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.724379 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:41:26.724707 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.724690 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:41:26.807752 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.561614 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:26.828275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.812753 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:26.828554 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.828514 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:26.828786 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.828767 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:41:26.865972 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.865931 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:26.866590 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.866545 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:41:26.913683 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.866911 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:26.913940 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:26.890351 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 17:41:27.001809 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.001693 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 17:41:27.002481 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.002024 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:27.064904 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.064869 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:41:27.125280 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.121514 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:41:27.287495 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.287400 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 17:41:27.314286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.291321 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 17:41:27.314286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.300528 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" event={"ID":"495ff2b5-327a-424e-a596-9c2b3dbfaf74","Type":"ContainerStarted","Data":"bebec3cf81d61f9987e95b7d5f07cb1c352704a8e4dd768fd402bcb6ffc6b229"} Oct 31 17:41:27.314286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.309334 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:27.314286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.309428 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 17:41:27.399721 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.399682 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 17:41:27.421978 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.421938 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 17:41:27.428410 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.428366 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:27.428686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.428613 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:27.499051 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.499005 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:27.523491 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.523453 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:27.625009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.602164 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:27.630333 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.626823 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:27.813464 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.813412 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 17:41:27.882649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.882548 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:27.886925 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.886886 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:27.914894 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.914847 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:27.965108 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:27.965062 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": context deadline exceeded" Oct 31 17:41:28.005419 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.005370 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:28.018712 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.018670 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:41:28.018989 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.018958 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:28.078088 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.078039 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:28.121442 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.121392 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:28.306861 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.306826 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 17:41:28.579241 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.579141 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 17:41:28.580121 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.356831 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:28.582110 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.356795 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:28.583209 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.561179 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 17:41:28.583392 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.583366 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 17:41:28.583617 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.563400 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:28.585193 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.526381 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:28.812291 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.541763 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:28.812962 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.542143 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 17:41:28.894328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.893443 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 17:41:28.894328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.894120 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 17:41:28.920707 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:28.920428 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:29.012634 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.012585 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:29.042651 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.042610 7618 generic.go:356] "Generic (PLEG): container finished" podID="495ff2b5-327a-424e-a596-9c2b3dbfaf74" containerID="bebec3cf81d61f9987e95b7d5f07cb1c352704a8e4dd768fd402bcb6ffc6b229" exitCode=0 Oct 31 17:41:29.105592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.104364 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:29.124298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.106483 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" event={"ID":"495ff2b5-327a-424e-a596-9c2b3dbfaf74","Type":"ContainerDied","Data":"bebec3cf81d61f9987e95b7d5f07cb1c352704a8e4dd768fd402bcb6ffc6b229"} Oct 31 17:41:29.124298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.106640 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:41:29.124298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.106715 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:41:29.125776 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.124782 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:29.129297 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.126216 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:41:29.229058 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.228949 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:41:29.229058 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.229028 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:29.346278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.337973 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 17:41:29.346278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.338032 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 17:41:29.388736 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.388698 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 17:41:29.389413 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.389125 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 17:41:29.573337 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.564294 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:41:29.573337 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.564377 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:29.595387 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.595337 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 31 17:41:29.603609 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.602093 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 17:41:29.604102 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.602138 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 17:41:29.604301 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.602365 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:41:29.604401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.604333 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:29.668279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.660569 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:29.807277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.803237 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:29.848338 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.826753 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:29.848338 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.826797 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 17:41:29.908436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.908380 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:29.909272 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.908760 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 17:41:29.947117 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:29.947071 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 17:41:30.017300 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.015487 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:30.033008 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.031460 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:41:30.033008 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.031523 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:30.138754 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.138636 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:30.165278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.155630 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:30.291442 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.291394 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:30.312542 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.312498 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:30.580455 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.580410 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 31 17:41:30.590284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.580822 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 17:41:30.590284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.580879 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 17:41:30.590284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.580930 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:41:30.590284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.580950 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:30.609405 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.608768 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:30.613276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.610528 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:30.613276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.610833 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 31 17:41:30.674323 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.673905 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 17:41:30.891340 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.866837 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 17:41:30.891340 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.867182 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 17:41:30.891340 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.868204 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 17:41:30.925621 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.920733 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:30.925621 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.921094 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 17:41:30.979280 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:30.977816 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:31.023276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.022467 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:31.246466 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.246408 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 17:41:31.413326 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.258594 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:41:31.421953 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.413712 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:31.490282 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.362087 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:41:31.502396 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.502349 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:31.842171 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.398840 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 17:41:31.843074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.843041 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:31.843347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.843319 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:41:32.144977 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.144947 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp"] Oct 31 17:41:32.194290 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.183819 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:32.197307 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.939031 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 17:41:32.201751 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.939066 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 17:41:32.204435 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.204412 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.204689 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.940738 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:32.205426 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.940553 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 31 17:41:32.223803 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.223773 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:32.232426 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.232391 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp"] Oct 31 17:41:32.232632 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.107700 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 17:41:32.235505 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.940680 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 17:41:32.246856 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:31.940704 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 17:41:32.260332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.089381 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:32.279195 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.101173 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:41:32.279496 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.279463 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:32.279631 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.106076 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:32.280329 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.107729 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 17:41:32.281488 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.107754 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:32.314342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.313791 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Oct 31 17:41:32.314342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.314111 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:41:32.366427 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.350291 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljsvs\" (UniqueName: \"kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-kube-api-access-ljsvs\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.420328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.395496 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-credential-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.443553 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.425421 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:32.449368 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.364609 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 31 17:41:32.458204 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.443246 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a1b216-1644-4e44-ae2c-4e23c06d7940-istiod-ca-cert\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.496512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.484267 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-certs\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.496512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.484413 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.496512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.484636 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-envoy\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.496512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.484679 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-data\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.496512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.484784 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-podinfo\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.496512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.484902 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-token\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.596274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.588209 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-token\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.596274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.588397 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljsvs\" (UniqueName: \"kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-kube-api-access-ljsvs\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.596274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.588453 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-credential-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.596274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.588484 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a1b216-1644-4e44-ae2c-4e23c06d7940-istiod-ca-cert\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.596274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.588517 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-certs\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.596274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.588598 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.596274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.588720 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-envoy\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.596274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.588753 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-data\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.596274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.588846 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-podinfo\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.616314 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.614941 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-credential-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.616314 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.615573 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a1b216-1644-4e44-ae2c-4e23c06d7940-istiod-ca-cert\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.616314 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.615788 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-certs\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.616314 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.615991 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.704231 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.704153 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-data\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.765272 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.763065 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-envoy\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.781375 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.775515 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljsvs\" (UniqueName: \"kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-kube-api-access-ljsvs\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.781375 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.775952 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-token\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.803788 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.803747 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-podinfo\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:32.848914 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.848869 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 17:41:32.983634 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.978052 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:32.992927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.992895 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:32.993242 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.993223 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:32.993837 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.993819 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:41:32.994031 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.994018 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:41:32.994236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.994221 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:41:32.994458 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.994429 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:41:32.994740 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.994727 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:41:32.995054 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.995037 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" Oct 31 17:41:32.995347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.995331 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:41:32.995513 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.995501 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:41:32.995626 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.995607 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd" event={"ID":"495ff2b5-327a-424e-a596-9c2b3dbfaf74","Type":"ContainerDied","Data":"09b5462368f323089575c9bac1c058223b65707132d258d23088bb553c3f00f9"} Oct 31 17:41:32.995761 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.995745 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09b5462368f323089575c9bac1c058223b65707132d258d23088bb553c3f00f9" Oct 31 17:41:32.997975 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.997955 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:41:32.998219 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.998202 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:41:32.998712 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.998688 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:41:32.998832 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.998814 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:41:32.998918 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.998904 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:41:32.999011 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:32.998999 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:41:33.027581 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.027555 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:41:33.057051 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.057007 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495ff2b5-327a-424e-a596-9c2b3dbfaf74-kube-api-access-s9c74\" (UniqueName: \"kubernetes.io/projected/495ff2b5-327a-424e-a596-9c2b3dbfaf74-kube-api-access-s9c74\") pod \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\" (UID: \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\") " Oct 31 17:41:33.057221 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.057073 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-openshift-ca\") pod \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\" (UID: \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\") " Oct 31 17:41:33.057221 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.057116 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-xks-ca\") pod \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\" (UID: \"495ff2b5-327a-424e-a596-9c2b3dbfaf74\") " Oct 31 17:41:33.074893 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.074854 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-openshift-ca" pod "495ff2b5-327a-424e-a596-9c2b3dbfaf74" (UID: "495ff2b5-327a-424e-a596-9c2b3dbfaf74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:41:33.102093 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.099854 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495ff2b5-327a-424e-a596-9c2b3dbfaf74-kube-api-access-s9c74" pod "495ff2b5-327a-424e-a596-9c2b3dbfaf74" (UID: "495ff2b5-327a-424e-a596-9c2b3dbfaf74"). InnerVolumeSpecName "kube-api-access-s9c74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:41:33.104900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.104865 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-xks-ca" pod "495ff2b5-327a-424e-a596-9c2b3dbfaf74" (UID: "495ff2b5-327a-424e-a596-9c2b3dbfaf74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:41:33.112920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.112892 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:33.161856 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.161817 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9c74\" (UniqueName: \"kubernetes.io/projected/495ff2b5-327a-424e-a596-9c2b3dbfaf74-kube-api-access-s9c74\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:41:33.161973 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.161859 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:41:33.161973 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.161876 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/495ff2b5-327a-424e-a596-9c2b3dbfaf74-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:41:33.540062 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.539944 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp"] Oct 31 17:41:33.569287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.568762 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:33.569287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.568855 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:33.569287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:33.568907 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:34.004330 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:34.004286 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" event={"ID":"82a1b216-1644-4e44-ae2c-4e23c06d7940","Type":"ContainerStarted","Data":"53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3"} Oct 31 17:41:34.004794 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:34.004357 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" event={"ID":"82a1b216-1644-4e44-ae2c-4e23c06d7940","Type":"ContainerStarted","Data":"49646a4e94464caa27209373770060195e106cebad7f78a4e57a3cf7944afabe"} Oct 31 17:41:34.042756 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:34.042694 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" podStartSLOduration=3.042675929 podStartE2EDuration="3.042675929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:34.042383004 +0000 UTC m=+815.790357473" watchObservedRunningTime="2026-10-31 17:41:34.042675929 +0000 UTC m=+815.790650397" Oct 31 17:41:34.114148 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:34.114058 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:34.115644 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:34.115612 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 17:41:34.115773 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:34.115671 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 17:41:35.114797 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:35.114731 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 17:41:35.115387 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:35.114843 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 17:41:36.113883 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:36.113717 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 17:41:36.113883 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:36.113786 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 17:41:37.113487 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:37.113452 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 17:41:37.113880 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:37.113503 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 17:41:38.114320 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:38.114285 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 17:41:38.114833 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:38.114356 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 17:41:38.580353 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:41:38.580312 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:41:39.114228 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:39.114192 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 17:41:39.114422 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:39.114279 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 17:41:40.113938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.113847 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 17:41:40.113938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.113923 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 17:41:40.455221 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.455173 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm"] Oct 31 17:41:40.455886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.455866 7618 cpu_manager.go:414] "containerMap: removing container" podUID="495ff2b5-327a-424e-a596-9c2b3dbfaf74" containerName="revoke-keys" Oct 31 17:41:40.455886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.455887 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495ff2b5-327a-424e-a596-9c2b3dbfaf74" containerName="revoke-keys" Oct 31 17:41:40.456070 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.456057 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="495ff2b5-327a-424e-a596-9c2b3dbfaf74" containerName="revoke-keys" Oct 31 17:41:40.461438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.461413 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:40.477508 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.477475 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm"] Oct 31 17:41:40.489238 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.489208 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:41:40.489436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.489362 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:41:40.517381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.517324 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69dc2860-qndhm\" (UID: \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:40.517381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.517380 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpqn\" (UniqueName: \"kubernetes.io/projected/0de3ccc3-7acf-451f-a5a9-905a474d1a68-kube-api-access-nqpqn\") pod \"maas-api-revoke-sub-69dc2860-qndhm\" (UID: \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:40.517713 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.517550 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69dc2860-qndhm\" (UID: \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:40.619734 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.619691 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69dc2860-qndhm\" (UID: \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:40.620007 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.619984 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69dc2860-qndhm\" (UID: \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:40.620068 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.620026 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpqn\" (UniqueName: \"kubernetes.io/projected/0de3ccc3-7acf-451f-a5a9-905a474d1a68-kube-api-access-nqpqn\") pod \"maas-api-revoke-sub-69dc2860-qndhm\" (UID: \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:40.621775 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.621746 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69dc2860-qndhm\" (UID: \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:40.622861 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.622837 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69dc2860-qndhm\" (UID: \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:40.637553 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.637510 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpqn\" (UniqueName: \"kubernetes.io/projected/0de3ccc3-7acf-451f-a5a9-905a474d1a68-kube-api-access-nqpqn\") pod \"maas-api-revoke-sub-69dc2860-qndhm\" (UID: \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:40.777353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:40.777312 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:41.095044 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:41:41.095010 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de3ccc3_7acf_451f_a5a9_905a474d1a68.slice/crio-e05a7a4bcad82a7e72117ca9424b00dd2ec2215b6ab849a7ea63400f083f0ccb WatchSource:0}: Error finding container e05a7a4bcad82a7e72117ca9424b00dd2ec2215b6ab849a7ea63400f083f0ccb: Status 404 returned error can't find the container with id e05a7a4bcad82a7e72117ca9424b00dd2ec2215b6ab849a7ea63400f083f0ccb Oct 31 17:41:41.104515 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:41.104488 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm"] Oct 31 17:41:41.115038 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:41.113330 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" event={"ID":"0de3ccc3-7acf-451f-a5a9-905a474d1a68","Type":"ContainerStarted","Data":"e05a7a4bcad82a7e72117ca9424b00dd2ec2215b6ab849a7ea63400f083f0ccb"} Oct 31 17:41:41.487454 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:41.487419 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:41.488312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:41.488289 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:41.489876 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:41.489854 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:41:42.121769 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:42.121732 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" event={"ID":"0de3ccc3-7acf-451f-a5a9-905a474d1a68","Type":"ContainerStarted","Data":"be134a2d68c48bf0cb84a9595b597122db77498aa139512c2239f48532b6deef"} Oct 31 17:41:42.172516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:42.172455 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" podStartSLOduration=2.172386638 podStartE2EDuration="2.172386638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:42.159624595 +0000 UTC m=+823.907599063" watchObservedRunningTime="2026-10-31 17:41:42.172386638 +0000 UTC m=+823.920361106" Oct 31 17:41:43.130870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:43.130831 7618 generic.go:356] "Generic (PLEG): container finished" podID="0de3ccc3-7acf-451f-a5a9-905a474d1a68" containerID="be134a2d68c48bf0cb84a9595b597122db77498aa139512c2239f48532b6deef" exitCode=0 Oct 31 17:41:43.131450 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:43.131162 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" event={"ID":"0de3ccc3-7acf-451f-a5a9-905a474d1a68","Type":"ContainerDied","Data":"be134a2d68c48bf0cb84a9595b597122db77498aa139512c2239f48532b6deef"} Oct 31 17:41:44.358918 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:44.358889 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:44.501452 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:44.501373 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-openshift-ca\") pod \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\" (UID: \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\") " Oct 31 17:41:44.501659 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:44.501644 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-xks-ca\") pod \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\" (UID: \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\") " Oct 31 17:41:44.501703 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:44.501678 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de3ccc3-7acf-451f-a5a9-905a474d1a68-kube-api-access-nqpqn\" (UniqueName: \"kubernetes.io/projected/0de3ccc3-7acf-451f-a5a9-905a474d1a68-kube-api-access-nqpqn\") pod \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\" (UID: \"0de3ccc3-7acf-451f-a5a9-905a474d1a68\") " Oct 31 17:41:44.501748 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:44.501696 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-openshift-ca" pod "0de3ccc3-7acf-451f-a5a9-905a474d1a68" (UID: "0de3ccc3-7acf-451f-a5a9-905a474d1a68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:41:44.502097 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:44.502070 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:41:44.503719 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:44.503694 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-xks-ca" pod "0de3ccc3-7acf-451f-a5a9-905a474d1a68" (UID: "0de3ccc3-7acf-451f-a5a9-905a474d1a68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:41:44.503978 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:44.503957 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de3ccc3-7acf-451f-a5a9-905a474d1a68-kube-api-access-nqpqn" pod "0de3ccc3-7acf-451f-a5a9-905a474d1a68" (UID: "0de3ccc3-7acf-451f-a5a9-905a474d1a68"). InnerVolumeSpecName "kube-api-access-nqpqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:41:44.608457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:44.608116 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de3ccc3-7acf-451f-a5a9-905a474d1a68-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:41:44.608457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:44.608173 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqpqn\" (UniqueName: \"kubernetes.io/projected/0de3ccc3-7acf-451f-a5a9-905a474d1a68-kube-api-access-nqpqn\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:41:45.147989 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:45.147962 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" Oct 31 17:41:45.148163 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:45.147983 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm" event={"ID":"0de3ccc3-7acf-451f-a5a9-905a474d1a68","Type":"ContainerDied","Data":"e05a7a4bcad82a7e72117ca9424b00dd2ec2215b6ab849a7ea63400f083f0ccb"} Oct 31 17:41:45.148163 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:45.148058 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e05a7a4bcad82a7e72117ca9424b00dd2ec2215b6ab849a7ea63400f083f0ccb" Oct 31 17:41:46.544153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.544109 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv"] Oct 31 17:41:46.545321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.545292 7618 cpu_manager.go:414] "containerMap: removing container" podUID="0de3ccc3-7acf-451f-a5a9-905a474d1a68" containerName="revoke-keys" Oct 31 17:41:46.545321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.545319 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de3ccc3-7acf-451f-a5a9-905a474d1a68" containerName="revoke-keys" Oct 31 17:41:46.545564 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.545550 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de3ccc3-7acf-451f-a5a9-905a474d1a68" containerName="revoke-keys" Oct 31 17:41:46.549757 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.549736 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.570374 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.570344 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg"] Oct 31 17:41:46.576516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.576493 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.580133 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.580103 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv"] Oct 31 17:41:46.610047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.610005 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg"] Oct 31 17:41:46.639648 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.639608 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-credential-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.639976 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.639912 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-data\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.640089 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.639984 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4vwj\" (UniqueName: \"kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-kube-api-access-b4vwj\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.640089 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640029 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79109729-f8dc-44d2-9366-128216bc2eb6-istio-podinfo\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.640089 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640056 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-istio-token\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.640316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640143 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-credential-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.640316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640204 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd011f8-7a9d-4814-871b-8349744550a5-istio-podinfo\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.640316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640274 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd011f8-7a9d-4814-871b-8349744550a5-istiod-ca-cert\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.640492 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640327 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-envoy\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.640492 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640352 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-envoy\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.640492 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640374 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-istio-token\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.640492 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640414 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79109729-f8dc-44d2-9366-128216bc2eb6-istiod-ca-cert\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.640686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640511 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-certs\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.640686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640542 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.640686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640569 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.640853 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.640833 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-certs\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.641024 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.641006 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ngdp\" (UniqueName: \"kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-kube-api-access-8ngdp\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.641177 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.641157 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-data\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.743151 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743113 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-credential-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.743382 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743204 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd011f8-7a9d-4814-871b-8349744550a5-istio-podinfo\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.743382 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743280 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd011f8-7a9d-4814-871b-8349744550a5-istiod-ca-cert\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.743382 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743331 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-envoy\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.743382 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743358 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-envoy\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.743598 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743380 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-istio-token\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.743598 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743421 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79109729-f8dc-44d2-9366-128216bc2eb6-istiod-ca-cert\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.743598 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743526 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-certs\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.743598 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743557 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.743598 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743589 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.743846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743641 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-certs\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.743846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743673 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ngdp\" (UniqueName: \"kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-kube-api-access-8ngdp\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.743846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743693 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-credential-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.743846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743703 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-data\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.743846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743769 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-credential-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.743846 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743833 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-data\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.744155 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743875 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vwj\" (UniqueName: \"kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-kube-api-access-b4vwj\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.744155 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743920 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79109729-f8dc-44d2-9366-128216bc2eb6-istio-podinfo\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.744155 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743948 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-istio-token\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.744155 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.743991 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-data\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.744155 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.744017 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd011f8-7a9d-4814-871b-8349744550a5-istiod-ca-cert\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.744478 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.744284 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-data\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.745118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.744686 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-credential-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.745118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.744753 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.745118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.744951 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-certs\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.745118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.745012 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-certs\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.745118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.745040 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79109729-f8dc-44d2-9366-128216bc2eb6-istiod-ca-cert\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.745406 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.745179 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-socket\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.746396 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.746368 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd011f8-7a9d-4814-871b-8349744550a5-istio-podinfo\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.746543 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.746525 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-envoy\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.746772 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.746756 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79109729-f8dc-44d2-9366-128216bc2eb6-istio-podinfo\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.747537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.747516 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-envoy\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.761184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.761154 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-istio-token\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.761373 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.761209 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ngdp\" (UniqueName: \"kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-kube-api-access-8ngdp\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.761655 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.761632 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-istio-token\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.761778 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.761754 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4vwj\" (UniqueName: \"kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-kube-api-access-b4vwj\") pod \"e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.831327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.831224 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw"] Oct 31 17:41:46.837098 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.837065 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.845732 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.845702 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzxl7\" (UniqueName: \"kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-kube-api-access-rzxl7\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.845886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.845755 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.845934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.845874 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-envoy\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.846165 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.846128 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-credential-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.846300 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.846246 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-podinfo\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.846420 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.846373 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-token\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.846483 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.846418 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-data\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.846525 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.846512 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70d0930-55ba-4c3d-9b94-046606c04bf0-istiod-ca-cert\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.846614 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.846598 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-certs\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.857959 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.857930 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw"] Oct 31 17:41:46.867565 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.867532 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:46.918868 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.918641 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:46.949002 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.948907 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-credential-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.949002 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.948998 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-podinfo\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.951693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.949034 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-token\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.951693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.949062 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-data\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.951693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.949095 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70d0930-55ba-4c3d-9b94-046606c04bf0-istiod-ca-cert\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.951693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.949193 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-certs\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.951693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.949969 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-certs\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.951693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.950000 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzxl7\" (UniqueName: \"kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-kube-api-access-rzxl7\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.951693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.950517 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.951693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.950609 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-envoy\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.951693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.950656 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70d0930-55ba-4c3d-9b94-046606c04bf0-istiod-ca-cert\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.951693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.950902 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.951693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.951160 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-data\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.951693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.951393 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-credential-socket\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.954473 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.954165 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-envoy\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.964827 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.964660 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-podinfo\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.967890 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.967829 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-token\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:46.970157 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:46.970096 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzxl7\" (UniqueName: \"kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-kube-api-access-rzxl7\") pod \"e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:47.155402 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.155359 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:47.249827 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.249790 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv"] Oct 31 17:41:47.274398 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.271525 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:47.274398 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.271614 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:47.274398 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.271656 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:47.292891 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.292833 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg"] Oct 31 17:41:47.307113 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:41:47.307061 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd011f8_7a9d_4814_871b_8349744550a5.slice/crio-afa71151a330d2e4f0c7a3671607236e20e1763cfb61b66c6590a485743cfb15 WatchSource:0}: Error finding container afa71151a330d2e4f0c7a3671607236e20e1763cfb61b66c6590a485743cfb15: Status 404 returned error can't find the container with id afa71151a330d2e4f0c7a3671607236e20e1763cfb61b66c6590a485743cfb15 Oct 31 17:41:47.310412 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.309980 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:47.310412 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.310065 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:47.310412 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.310104 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:47.487695 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.487639 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:47.487812 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.487728 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:47.487812 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.487754 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:41:47.494135 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:47.494110 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw"] Oct 31 17:41:48.203624 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.203579 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" event={"ID":"79109729-f8dc-44d2-9366-128216bc2eb6","Type":"ContainerStarted","Data":"e3bd2e57f25fd5c2681fb5e95ac088666da48ea6424acfdd1f556476d7b4bbbd"} Oct 31 17:41:48.204127 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.203653 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" event={"ID":"79109729-f8dc-44d2-9366-128216bc2eb6","Type":"ContainerStarted","Data":"4432772a9fcebe27041ee5d47cb86f2f3788afd6a099554e449d44f154abdd4c"} Oct 31 17:41:48.207469 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.207435 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" event={"ID":"f70d0930-55ba-4c3d-9b94-046606c04bf0","Type":"ContainerStarted","Data":"cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e"} Oct 31 17:41:48.207673 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.207497 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" event={"ID":"f70d0930-55ba-4c3d-9b94-046606c04bf0","Type":"ContainerStarted","Data":"b2ee0607e919008efa6be6a81f63a830f53a79c6cb8f274b3fbdd3ae0771fcc9"} Oct 31 17:41:48.209822 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.209788 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" event={"ID":"8bd011f8-7a9d-4814-871b-8349744550a5","Type":"ContainerStarted","Data":"73c514d6dac9e1b2ff0750f882859e26e9a6a3a7d6ec528d7f56b5827e8ab079"} Oct 31 17:41:48.209946 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.209835 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" event={"ID":"8bd011f8-7a9d-4814-871b-8349744550a5","Type":"ContainerStarted","Data":"afa71151a330d2e4f0c7a3671607236e20e1763cfb61b66c6590a485743cfb15"} Oct 31 17:41:48.328900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.328832 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" podStartSLOduration=2.32880864 podStartE2EDuration="2.32880864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:48.286632271 +0000 UTC m=+830.034606740" watchObservedRunningTime="2026-10-31 17:41:48.32880864 +0000 UTC m=+830.076783108" Oct 31 17:41:48.392946 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.387460 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" podStartSLOduration=2.387442917 podStartE2EDuration="2.387442917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:48.383497216 +0000 UTC m=+830.131471697" watchObservedRunningTime="2026-10-31 17:41:48.387442917 +0000 UTC m=+830.135417386" Oct 31 17:41:48.405840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.405782 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" podStartSLOduration=2.405747875 podStartE2EDuration="2.405747875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:48.344186749 +0000 UTC m=+830.092161218" watchObservedRunningTime="2026-10-31 17:41:48.405747875 +0000 UTC m=+830.153722342" Oct 31 17:41:48.873411 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.873379 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:48.873750 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.873727 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 17:41:48.873927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.873905 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 17:41:48.924097 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.924062 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:48.924514 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.924489 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 17:41:48.924699 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:48.924675 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 17:41:49.169345 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:49.161579 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:49.169345 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:49.162480 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 17:41:49.169345 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:49.162531 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 17:41:49.869278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:49.868664 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 17:41:49.869278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:49.868740 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 17:41:49.924278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:49.923761 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 17:41:49.924278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:49.923841 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 17:41:50.185277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:50.184588 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 17:41:50.185277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:50.184655 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 17:41:50.889278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:50.869783 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 17:41:51.013276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:50.920672 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 17:41:51.013276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.012510 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 17:41:51.038354 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:50.986757 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 17:41:51.171754 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.171541 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 17:41:51.171754 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.171601 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 17:41:51.231278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.227802 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2"] Oct 31 17:41:51.381305 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.293361 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:51.381305 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.339295 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 17:41:51.599903 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.595503 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2"] Oct 31 17:41:51.599903 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.596044 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:51.642695 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.638066 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:41:51.689316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.676694 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:41:51.711140 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.711028 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a26f47-a048-453c-8950-36c92b30520c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2\" (UID: \"88a26f47-a048-453c-8950-36c92b30520c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:51.711356 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.711280 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk9gv\" (UniqueName: \"kubernetes.io/projected/88a26f47-a048-453c-8950-36c92b30520c-kube-api-access-dk9gv\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2\" (UID: \"88a26f47-a048-453c-8950-36c92b30520c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:51.711454 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.711431 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a26f47-a048-453c-8950-36c92b30520c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2\" (UID: \"88a26f47-a048-453c-8950-36c92b30520c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:51.823388 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.818319 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a26f47-a048-453c-8950-36c92b30520c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2\" (UID: \"88a26f47-a048-453c-8950-36c92b30520c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:51.823734 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.823706 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9gv\" (UniqueName: \"kubernetes.io/projected/88a26f47-a048-453c-8950-36c92b30520c-kube-api-access-dk9gv\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2\" (UID: \"88a26f47-a048-453c-8950-36c92b30520c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:51.828387 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.823974 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a26f47-a048-453c-8950-36c92b30520c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2\" (UID: \"88a26f47-a048-453c-8950-36c92b30520c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:51.858907 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:51.822132 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a26f47-a048-453c-8950-36c92b30520c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2\" (UID: \"88a26f47-a048-453c-8950-36c92b30520c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:52.016281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.012727 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a26f47-a048-453c-8950-36c92b30520c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2\" (UID: \"88a26f47-a048-453c-8950-36c92b30520c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:52.046238 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.013573 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 17:41:52.046562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.046516 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 17:41:52.047140 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.040340 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Oct 31 17:41:52.047322 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.013608 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 17:41:52.047460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.047441 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 17:41:52.101287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.099597 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk9gv\" (UniqueName: \"kubernetes.io/projected/88a26f47-a048-453c-8950-36c92b30520c-kube-api-access-dk9gv\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2\" (UID: \"88a26f47-a048-453c-8950-36c92b30520c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:52.159453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.159417 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 17:41:52.159802 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.159769 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 17:41:52.291277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.279575 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:52.414337 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.414124 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:52.422309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.415025 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:41:52.605276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.585449 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:52.605276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.585323 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:52.605276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.586750 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 31 17:41:52.824516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.824423 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:52.842285 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.838642 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:52.859396 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.859347 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 17:41:52.900626 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.900569 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 17:41:52.901438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.901105 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 17:41:52.909503 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.909450 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 17:41:52.911342 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.911310 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 17:41:52.933640 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.933606 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 17:41:52.933957 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.933926 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 17:41:52.966624 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:52.966585 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:53.011948 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.011909 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 17:41:53.041699 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.041654 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:53.201516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.201478 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 17:41:53.232080 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.230445 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 17:41:53.232080 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.230350 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:53.496882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.496369 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:53.497288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.497138 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 17:41:53.617463 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.614382 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:53.617463 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.615760 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:41:53.638857 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.635404 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:53.638857 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.635591 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:41:53.638857 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.636051 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:53.640218 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.639422 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:41:53.833391 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.833292 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 17:41:53.878428 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.878378 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:53.939748 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.879746 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:41:53.940092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.940073 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:41:53.955281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.954638 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 17:41:53.955281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.954708 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 17:41:53.969309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.964354 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:53.998353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.969586 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:41:53.998353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.970635 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 17:41:53.998353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:53.971183 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:41:54.031902 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.031867 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 17:41:54.032163 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.032129 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 17:41:54.105160 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.078517 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:54.105160 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.079391 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:41:54.105160 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.082327 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 17:41:54.173076 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.136484 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:41:54.244568 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.244534 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 17:41:54.244834 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.244807 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 17:41:54.340302 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.335490 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:41:54.340302 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.335561 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:41:54.650645 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.650599 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 17:41:54.651394 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.651363 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 17:41:54.654265 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.654218 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 31 17:41:54.682216 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.682045 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:54.701477 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.701433 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:41:54.775274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.768427 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:41:54.775274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.768505 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:41:54.775274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.768544 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:41:54.775274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.768565 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2"] Oct 31 17:41:54.775274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.768622 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:41:54.775274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.768655 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:41:54.775274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.772003 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:41:54.800293 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.798163 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:41:54.813287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.812438 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:41:54.839140 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.837459 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:41:54.839140 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.838591 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:41:54.869891 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.869859 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 17:41:54.870124 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.870097 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 17:41:54.961232 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.960638 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 17:41:54.961232 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:54.960702 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 17:41:55.156047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:55.156012 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 17:41:55.156608 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:55.156575 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 17:41:55.459428 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:55.459385 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk"] Oct 31 17:41:55.696284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:55.693821 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" event={"ID":"88a26f47-a048-453c-8950-36c92b30520c","Type":"ContainerStarted","Data":"5c7a1d36acfc03b271d7f02591f37ad9b0438ab4b7d9715729cb7995c7a68983"} Oct 31 17:41:55.696284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:55.693892 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" event={"ID":"88a26f47-a048-453c-8950-36c92b30520c","Type":"ContainerStarted","Data":"e02b55879035397a4b9a3ccfce2f62d03e9cfc319eae353681effff04e69d690"} Oct 31 17:41:55.739354 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:55.738389 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" podStartSLOduration=5.73836806 podStartE2EDuration="5.73836806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:55.731136118 +0000 UTC m=+837.479110586" watchObservedRunningTime="2026-10-31 17:41:55.73836806 +0000 UTC m=+837.486342538" Oct 31 17:41:55.870328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:55.869829 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 31 17:41:55.870328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:55.869898 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 31 17:41:56.332517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:56.332483 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:56.334311 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:56.334289 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:56.348482 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:56.348449 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:41:56.734782 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:56.734750 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:56.741283 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:56.740631 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:56.744280 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:56.741920 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:41:56.766004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:56.765967 7618 generic.go:356] "Generic (PLEG): container finished" podID="88a26f47-a048-453c-8950-36c92b30520c" containerID="5c7a1d36acfc03b271d7f02591f37ad9b0438ab4b7d9715729cb7995c7a68983" exitCode=0 Oct 31 17:41:56.767527 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:56.767499 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" event={"ID":"88a26f47-a048-453c-8950-36c92b30520c","Type":"ContainerDied","Data":"5c7a1d36acfc03b271d7f02591f37ad9b0438ab4b7d9715729cb7995c7a68983"} Oct 31 17:41:56.874546 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:56.874512 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:56.875455 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:56.875427 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:56.876788 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:56.876760 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:41:57.298364 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.298329 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c"] Oct 31 17:41:57.305588 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.305557 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:41:57.320365 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.320319 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c"] Oct 31 17:41:57.366990 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.366950 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d83c999-2xm7c\" (UID: \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:41:57.367402 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.367023 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrpbs\" (UniqueName: \"kubernetes.io/projected/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-kube-api-access-xrpbs\") pod \"maas-api-revoke-sub-5d83c999-2xm7c\" (UID: \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:41:57.367402 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.367146 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d83c999-2xm7c\" (UID: \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:41:57.468658 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.468619 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d83c999-2xm7c\" (UID: \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:41:57.468848 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.468769 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrpbs\" (UniqueName: \"kubernetes.io/projected/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-kube-api-access-xrpbs\") pod \"maas-api-revoke-sub-5d83c999-2xm7c\" (UID: \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:41:57.468848 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.468822 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d83c999-2xm7c\" (UID: \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:41:57.469563 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.469537 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d83c999-2xm7c\" (UID: \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:41:57.471190 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.471171 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d83c999-2xm7c\" (UID: \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:41:57.485704 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.485674 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrpbs\" (UniqueName: \"kubernetes.io/projected/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-kube-api-access-xrpbs\") pod \"maas-api-revoke-sub-5d83c999-2xm7c\" (UID: \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:41:57.632385 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.632267 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:41:57.972595 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.972565 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c"] Oct 31 17:41:57.972694 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:41:57.972636 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a06a816_c2a1_4e85_bf30_fc36ae674b7e.slice/crio-ed1091490cedb970af0b7c36b85f4d5dc1fca725de8ff399022875e328d936f9 WatchSource:0}: Error finding container ed1091490cedb970af0b7c36b85f4d5dc1fca725de8ff399022875e328d936f9: Status 404 returned error can't find the container with id ed1091490cedb970af0b7c36b85f4d5dc1fca725de8ff399022875e328d936f9 Oct 31 17:41:57.979704 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.979681 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:57.989212 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.989182 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a26f47-a048-453c-8950-36c92b30520c-kube-api-access-dk9gv\" (UniqueName: \"kubernetes.io/projected/88a26f47-a048-453c-8950-36c92b30520c-kube-api-access-dk9gv\") pod \"88a26f47-a048-453c-8950-36c92b30520c\" (UID: \"88a26f47-a048-453c-8950-36c92b30520c\") " Oct 31 17:41:57.989361 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.989320 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a26f47-a048-453c-8950-36c92b30520c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a26f47-a048-453c-8950-36c92b30520c-maas-api-openshift-ca\") pod \"88a26f47-a048-453c-8950-36c92b30520c\" (UID: \"88a26f47-a048-453c-8950-36c92b30520c\") " Oct 31 17:41:57.989547 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.989531 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88a26f47-a048-453c-8950-36c92b30520c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a26f47-a048-453c-8950-36c92b30520c-maas-api-xks-ca\") pod \"88a26f47-a048-453c-8950-36c92b30520c\" (UID: \"88a26f47-a048-453c-8950-36c92b30520c\") " Oct 31 17:41:57.989730 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.989705 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a26f47-a048-453c-8950-36c92b30520c-maas-api-openshift-ca" pod "88a26f47-a048-453c-8950-36c92b30520c" (UID: "88a26f47-a048-453c-8950-36c92b30520c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:41:57.990241 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.990221 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a26f47-a048-453c-8950-36c92b30520c-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:41:57.991870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.991845 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a26f47-a048-453c-8950-36c92b30520c-kube-api-access-dk9gv" pod "88a26f47-a048-453c-8950-36c92b30520c" (UID: "88a26f47-a048-453c-8950-36c92b30520c"). InnerVolumeSpecName "kube-api-access-dk9gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:41:57.992137 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:57.992107 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a26f47-a048-453c-8950-36c92b30520c-maas-api-xks-ca" pod "88a26f47-a048-453c-8950-36c92b30520c" (UID: "88a26f47-a048-453c-8950-36c92b30520c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:41:58.091728 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.091667 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk9gv\" (UniqueName: \"kubernetes.io/projected/88a26f47-a048-453c-8950-36c92b30520c-kube-api-access-dk9gv\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:41:58.091728 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.091707 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a26f47-a048-453c-8950-36c92b30520c-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:41:58.323232 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.323198 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt"] Oct 31 17:41:58.324419 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.324395 7618 cpu_manager.go:414] "containerMap: removing container" podUID="88a26f47-a048-453c-8950-36c92b30520c" containerName="revoke-keys" Oct 31 17:41:58.324482 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.324423 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a26f47-a048-453c-8950-36c92b30520c" containerName="revoke-keys" Oct 31 17:41:58.324630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.324618 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a26f47-a048-453c-8950-36c92b30520c" containerName="revoke-keys" Oct 31 17:41:58.330367 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.330329 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:41:58.344607 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.344576 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt"] Oct 31 17:41:58.397491 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.396093 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt\" (UID: \"67eefead-6f47-43da-88ca-3b5e2780e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:41:58.397491 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.396220 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt\" (UID: \"67eefead-6f47-43da-88ca-3b5e2780e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:41:58.397491 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.396418 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxff\" (UniqueName: \"kubernetes.io/projected/67eefead-6f47-43da-88ca-3b5e2780e39e-kube-api-access-srxff\") pod \"maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt\" (UID: \"67eefead-6f47-43da-88ca-3b5e2780e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:41:58.506985 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.506944 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt\" (UID: \"67eefead-6f47-43da-88ca-3b5e2780e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:41:58.507185 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.507063 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt\" (UID: \"67eefead-6f47-43da-88ca-3b5e2780e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:41:58.507293 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.507241 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxff\" (UniqueName: \"kubernetes.io/projected/67eefead-6f47-43da-88ca-3b5e2780e39e-kube-api-access-srxff\") pod \"maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt\" (UID: \"67eefead-6f47-43da-88ca-3b5e2780e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:41:58.512567 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.512375 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt\" (UID: \"67eefead-6f47-43da-88ca-3b5e2780e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:41:58.514724 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.514664 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt\" (UID: \"67eefead-6f47-43da-88ca-3b5e2780e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:41:58.530994 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.530959 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxff\" (UniqueName: \"kubernetes.io/projected/67eefead-6f47-43da-88ca-3b5e2780e39e-kube-api-access-srxff\") pod \"maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt\" (UID: \"67eefead-6f47-43da-88ca-3b5e2780e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:41:58.658751 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.658712 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:41:58.803247 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.803204 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" event={"ID":"7a06a816-c2a1-4e85-bf30-fc36ae674b7e","Type":"ContainerStarted","Data":"b7a1c1d01617110cbd7e083bd7367785545012bcb2f8b7845c6ee35916e2394c"} Oct 31 17:41:58.803451 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.803322 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" event={"ID":"7a06a816-c2a1-4e85-bf30-fc36ae674b7e","Type":"ContainerStarted","Data":"ed1091490cedb970af0b7c36b85f4d5dc1fca725de8ff399022875e328d936f9"} Oct 31 17:41:58.807057 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.807025 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" Oct 31 17:41:58.807674 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.807650 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2" event={"ID":"88a26f47-a048-453c-8950-36c92b30520c","Type":"ContainerDied","Data":"e02b55879035397a4b9a3ccfce2f62d03e9cfc319eae353681effff04e69d690"} Oct 31 17:41:58.807822 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.807694 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e02b55879035397a4b9a3ccfce2f62d03e9cfc319eae353681effff04e69d690" Oct 31 17:41:58.841521 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.841463 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" podStartSLOduration=1.8414499009999998 podStartE2EDuration="1.841449901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:58.837567669 +0000 UTC m=+840.585542137" watchObservedRunningTime="2026-10-31 17:41:58.841449901 +0000 UTC m=+840.589424368" Oct 31 17:41:58.979855 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:58.979824 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt"] Oct 31 17:41:58.981031 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:41:58.981000 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67eefead_6f47_43da_88ca_3b5e2780e39e.slice/crio-5b05bd6d1a349af87c177cff6fb754a8e0a79da6c27ac5e196a4e0057b6209ab WatchSource:0}: Error finding container 5b05bd6d1a349af87c177cff6fb754a8e0a79da6c27ac5e196a4e0057b6209ab: Status 404 returned error can't find the container with id 5b05bd6d1a349af87c177cff6fb754a8e0a79da6c27ac5e196a4e0057b6209ab Oct 31 17:41:59.816996 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:59.816903 7618 generic.go:356] "Generic (PLEG): container finished" podID="7a06a816-c2a1-4e85-bf30-fc36ae674b7e" containerID="b7a1c1d01617110cbd7e083bd7367785545012bcb2f8b7845c6ee35916e2394c" exitCode=0 Oct 31 17:41:59.817423 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:59.816978 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" event={"ID":"7a06a816-c2a1-4e85-bf30-fc36ae674b7e","Type":"ContainerDied","Data":"b7a1c1d01617110cbd7e083bd7367785545012bcb2f8b7845c6ee35916e2394c"} Oct 31 17:41:59.819044 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:59.819014 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" event={"ID":"67eefead-6f47-43da-88ca-3b5e2780e39e","Type":"ContainerStarted","Data":"77a2b14e1e1d04e099cd1e6b1a8425960d2db0497f9a928223f6f00d8972fc8a"} Oct 31 17:41:59.819184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:59.819059 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" event={"ID":"67eefead-6f47-43da-88ca-3b5e2780e39e","Type":"ContainerStarted","Data":"5b05bd6d1a349af87c177cff6fb754a8e0a79da6c27ac5e196a4e0057b6209ab"} Oct 31 17:41:59.886609 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:41:59.886544 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" podStartSLOduration=1.8865241830000001 podStartE2EDuration="1.886524183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:41:59.883464432 +0000 UTC m=+841.631438899" watchObservedRunningTime="2026-10-31 17:41:59.886524183 +0000 UTC m=+841.634498650" Oct 31 17:42:00.829245 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:00.829209 7618 generic.go:356] "Generic (PLEG): container finished" podID="67eefead-6f47-43da-88ca-3b5e2780e39e" containerID="77a2b14e1e1d04e099cd1e6b1a8425960d2db0497f9a928223f6f00d8972fc8a" exitCode=0 Oct 31 17:42:00.829736 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:00.829288 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" event={"ID":"67eefead-6f47-43da-88ca-3b5e2780e39e","Type":"ContainerDied","Data":"77a2b14e1e1d04e099cd1e6b1a8425960d2db0497f9a928223f6f00d8972fc8a"} Oct 31 17:42:01.192316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.192287 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:42:01.276439 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.276394 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-openshift-ca\") pod \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\" (UID: \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\") " Oct 31 17:42:01.277159 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.277131 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-xks-ca\") pod \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\" (UID: \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\") " Oct 31 17:42:01.277548 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.277528 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-kube-api-access-xrpbs\" (UniqueName: \"kubernetes.io/projected/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-kube-api-access-xrpbs\") pod \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\" (UID: \"7a06a816-c2a1-4e85-bf30-fc36ae674b7e\") " Oct 31 17:42:01.299860 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.281284 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-kube-api-access-xrpbs" pod "7a06a816-c2a1-4e85-bf30-fc36ae674b7e" (UID: "7a06a816-c2a1-4e85-bf30-fc36ae674b7e"). InnerVolumeSpecName "kube-api-access-xrpbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:01.391483 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.390769 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrpbs\" (UniqueName: \"kubernetes.io/projected/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-kube-api-access-xrpbs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:01.392318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.392245 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-openshift-ca" pod "7a06a816-c2a1-4e85-bf30-fc36ae674b7e" (UID: "7a06a816-c2a1-4e85-bf30-fc36ae674b7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:01.394105 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.393395 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-xks-ca" pod "7a06a816-c2a1-4e85-bf30-fc36ae674b7e" (UID: "7a06a816-c2a1-4e85-bf30-fc36ae674b7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:01.496274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.494921 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:01.496274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.494977 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a06a816-c2a1-4e85-bf30-fc36ae674b7e-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:01.898351 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.890968 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" Oct 31 17:42:01.910378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.900775 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c" event={"ID":"7a06a816-c2a1-4e85-bf30-fc36ae674b7e","Type":"ContainerDied","Data":"ed1091490cedb970af0b7c36b85f4d5dc1fca725de8ff399022875e328d936f9"} Oct 31 17:42:01.910378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:01.900848 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed1091490cedb970af0b7c36b85f4d5dc1fca725de8ff399022875e328d936f9" Oct 31 17:42:02.930276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:02.923348 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:03.216712 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.170316 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz"] Oct 31 17:42:03.332516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.251445 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 17:42:03.332516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.304609 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 17:42:03.389279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.386572 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8"] Oct 31 17:42:03.438697 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.438245 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96"] Oct 31 17:42:03.492413 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.492298 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5"] Oct 31 17:42:03.496707 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.492700 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Oct 31 17:42:03.511299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.502703 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:03.706871 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.706558 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:03.713561 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.713507 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 17:42:03.723448 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.714479 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 31 17:42:03.731641 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.715063 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:03.810571 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.810489 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:42:03.944470 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.944419 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:03.985316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.978401 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:03.985316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:03.981196 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 17:42:04.119327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.111210 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:42:04.119327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.117464 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:04.200289 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.200144 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" event={"ID":"67eefead-6f47-43da-88ca-3b5e2780e39e","Type":"ContainerDied","Data":"5b05bd6d1a349af87c177cff6fb754a8e0a79da6c27ac5e196a4e0057b6209ab"} Oct 31 17:42:04.212087 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.212056 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b05bd6d1a349af87c177cff6fb754a8e0a79da6c27ac5e196a4e0057b6209ab" Oct 31 17:42:04.212436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.211479 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt" Oct 31 17:42:04.294478 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.294449 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67eefead-6f47-43da-88ca-3b5e2780e39e-kube-api-access-srxff\" (UniqueName: \"kubernetes.io/projected/67eefead-6f47-43da-88ca-3b5e2780e39e-kube-api-access-srxff\") pod \"67eefead-6f47-43da-88ca-3b5e2780e39e\" (UID: \"67eefead-6f47-43da-88ca-3b5e2780e39e\") " Oct 31 17:42:04.295005 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.294972 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-xks-ca\") pod \"67eefead-6f47-43da-88ca-3b5e2780e39e\" (UID: \"67eefead-6f47-43da-88ca-3b5e2780e39e\") " Oct 31 17:42:04.295321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.295304 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-openshift-ca\") pod \"67eefead-6f47-43da-88ca-3b5e2780e39e\" (UID: \"67eefead-6f47-43da-88ca-3b5e2780e39e\") " Oct 31 17:42:04.395100 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.297551 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:04.395944 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.319358 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 17:42:04.417356 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.359274 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:42:04.417686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.417664 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:42:04.420731 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.418053 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:42:04.522585 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.522548 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67eefead-6f47-43da-88ca-3b5e2780e39e-kube-api-access-srxff" pod "67eefead-6f47-43da-88ca-3b5e2780e39e" (UID: "67eefead-6f47-43da-88ca-3b5e2780e39e"). InnerVolumeSpecName "kube-api-access-srxff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:04.553643 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.553594 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:04.563626 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.563592 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:42:04.699355 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.698848 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srxff\" (UniqueName: \"kubernetes.io/projected/67eefead-6f47-43da-88ca-3b5e2780e39e-kube-api-access-srxff\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:04.754639 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.723871 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-xks-ca" pod "67eefead-6f47-43da-88ca-3b5e2780e39e" (UID: "67eefead-6f47-43da-88ca-3b5e2780e39e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:04.754639 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.724280 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 31 17:42:04.754639 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.724828 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:04.754639 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.748347 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:04.754639 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.748742 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 17:42:04.770101 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.770053 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-openshift-ca" pod "67eefead-6f47-43da-88ca-3b5e2780e39e" (UID: "67eefead-6f47-43da-88ca-3b5e2780e39e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:04.826497 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.826456 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:04.826497 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.826497 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67eefead-6f47-43da-88ca-3b5e2780e39e-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:04.981335 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:04.977360 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:05.027278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.012389 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:05.211195 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.211148 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 17:42:05.419388 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.412866 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:05.429574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.426182 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:05.429574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.428058 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:42:05.429574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.428115 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:42:05.429574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.428151 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:42:05.429574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.428189 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:42:05.429574 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.428223 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:42:05.615440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.615392 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 17:42:05.974020 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.805108 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 17:42:06.001766 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.807539 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 17:42:06.003614 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.808102 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 17:42:06.125656 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:06.033493 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 31 17:42:06.125656 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:06.079655 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:06.125656 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.899975 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 17:42:06.125656 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:06.096989 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:42:06.125656 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.899921 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 31 17:42:06.125656 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:05.998628 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 17:42:06.239945 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:06.239366 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:06.331630 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:06.331562 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:06.457853 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:06.457795 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:06.459534 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:06.458119 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:06.966377 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:06.966328 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:06.976097 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:06.976049 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 31 17:42:07.198133 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.013914 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 17:42:07.199362 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.095483 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 17:42:07.199821 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.098095 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:42:07.200043 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.200027 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:42:07.200177 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.200167 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:42:07.200394 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.200321 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:42:07.208434 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.208409 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:42:07.208698 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.208684 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:42:07.208832 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.208822 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:42:07.229357 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.225341 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:42:07.229572 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.138707 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 17:42:07.230118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.156110 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:07.354278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.299568 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:42:07.373288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.369529 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:07.392569 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.392541 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:42:07.395873 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.395769 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:42:07.395873 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.395838 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:42:07.396100 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.395888 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:42:07.396100 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.395922 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:42:07.418359 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:07.418330 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:42:08.170017 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.169979 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2"] Oct 31 17:42:08.170767 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.170748 7618 cpu_manager.go:414] "containerMap: removing container" podUID="67eefead-6f47-43da-88ca-3b5e2780e39e" containerName="revoke-keys" Oct 31 17:42:08.170837 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.170768 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67eefead-6f47-43da-88ca-3b5e2780e39e" containerName="revoke-keys" Oct 31 17:42:08.170894 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.170840 7618 cpu_manager.go:414] "containerMap: removing container" podUID="7a06a816-c2a1-4e85-bf30-fc36ae674b7e" containerName="revoke-keys" Oct 31 17:42:08.170894 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.170846 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a06a816-c2a1-4e85-bf30-fc36ae674b7e" containerName="revoke-keys" Oct 31 17:42:08.170961 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.170949 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a06a816-c2a1-4e85-bf30-fc36ae674b7e" containerName="revoke-keys" Oct 31 17:42:08.170994 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.170961 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="67eefead-6f47-43da-88ca-3b5e2780e39e" containerName="revoke-keys" Oct 31 17:42:08.175825 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.175804 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:08.188635 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.188603 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:42:08.189315 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.189292 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:42:08.191310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.191290 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2"] Oct 31 17:42:08.311009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.310965 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmbsk\" (UniqueName: \"kubernetes.io/projected/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-kube-api-access-pmbsk\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2\" (UID: \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:08.311491 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.311106 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2\" (UID: \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:08.311491 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.311145 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2\" (UID: \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:08.412721 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.412677 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmbsk\" (UniqueName: \"kubernetes.io/projected/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-kube-api-access-pmbsk\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2\" (UID: \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:08.412925 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.412838 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2\" (UID: \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:08.412925 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.412893 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2\" (UID: \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:08.413735 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.413706 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2\" (UID: \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:08.415641 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.415615 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2\" (UID: \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:08.430226 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.430156 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmbsk\" (UniqueName: \"kubernetes.io/projected/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-kube-api-access-pmbsk\") pod \"maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2\" (UID: \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:08.493465 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.493425 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:08.593033 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:08.592993 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:42:08.809596 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:42:08.809562 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ab7228_ad0c_4a75_b452_550e9db7bc8b.slice/crio-88cab59dcad1ccec0eafa7acd6a0aaf9ed77be9a7fc7c69816ccc27e1cd6d7d4 WatchSource:0}: Error finding container 88cab59dcad1ccec0eafa7acd6a0aaf9ed77be9a7fc7c69816ccc27e1cd6d7d4: Status 404 returned error can't find the container with id 88cab59dcad1ccec0eafa7acd6a0aaf9ed77be9a7fc7c69816ccc27e1cd6d7d4 Oct 31 17:42:08.810640 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.810613 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2"] Oct 31 17:42:08.895612 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.895580 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts"] Oct 31 17:42:08.901850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.901823 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:08.925315 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:08.925274 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts"] Oct 31 17:42:09.024097 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.024065 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5219a4-p69ts\" (UID: \"8ba87c0a-822f-467a-ab42-26cba2077357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:09.024387 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.024302 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cr92\" (UniqueName: \"kubernetes.io/projected/8ba87c0a-822f-467a-ab42-26cba2077357-kube-api-access-6cr92\") pod \"maas-api-revoke-sub-4f5219a4-p69ts\" (UID: \"8ba87c0a-822f-467a-ab42-26cba2077357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:09.024387 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.024355 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5219a4-p69ts\" (UID: \"8ba87c0a-822f-467a-ab42-26cba2077357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:09.125518 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.125427 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cr92\" (UniqueName: \"kubernetes.io/projected/8ba87c0a-822f-467a-ab42-26cba2077357-kube-api-access-6cr92\") pod \"maas-api-revoke-sub-4f5219a4-p69ts\" (UID: \"8ba87c0a-822f-467a-ab42-26cba2077357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:09.125518 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.125512 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5219a4-p69ts\" (UID: \"8ba87c0a-822f-467a-ab42-26cba2077357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:09.125751 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.125710 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5219a4-p69ts\" (UID: \"8ba87c0a-822f-467a-ab42-26cba2077357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:09.126391 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.126363 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5219a4-p69ts\" (UID: \"8ba87c0a-822f-467a-ab42-26cba2077357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:09.128167 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.128143 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5219a4-p69ts\" (UID: \"8ba87c0a-822f-467a-ab42-26cba2077357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:09.142891 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.142853 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cr92\" (UniqueName: \"kubernetes.io/projected/8ba87c0a-822f-467a-ab42-26cba2077357-kube-api-access-6cr92\") pod \"maas-api-revoke-sub-4f5219a4-p69ts\" (UID: \"8ba87c0a-822f-467a-ab42-26cba2077357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:09.222865 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.222830 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:09.298775 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.298742 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" event={"ID":"e1ab7228-ad0c-4a75-b452-550e9db7bc8b","Type":"ContainerStarted","Data":"fb54c822ac096aaa50ccd3ca1d70e970c1493a8cd2d72edd1b827fdffc5d9dc1"} Oct 31 17:42:09.298913 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.298796 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" event={"ID":"e1ab7228-ad0c-4a75-b452-550e9db7bc8b","Type":"ContainerStarted","Data":"88cab59dcad1ccec0eafa7acd6a0aaf9ed77be9a7fc7c69816ccc27e1cd6d7d4"} Oct 31 17:42:09.534691 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.534621 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" podStartSLOduration=1.534600034 podStartE2EDuration="1.534600034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:42:09.340674112 +0000 UTC m=+851.088648580" watchObservedRunningTime="2026-10-31 17:42:09.534600034 +0000 UTC m=+851.282574503" Oct 31 17:42:09.538493 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:09.538462 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts"] Oct 31 17:42:09.540545 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:42:09.540509 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba87c0a_822f_467a_ab42_26cba2077357.slice/crio-51e05e0ebdaac5171f0efd757d0d95346aee692adbe5edddee75787a9b5d453a WatchSource:0}: Error finding container 51e05e0ebdaac5171f0efd757d0d95346aee692adbe5edddee75787a9b5d453a: Status 404 returned error can't find the container with id 51e05e0ebdaac5171f0efd757d0d95346aee692adbe5edddee75787a9b5d453a Oct 31 17:42:10.311929 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:10.311819 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" event={"ID":"8ba87c0a-822f-467a-ab42-26cba2077357","Type":"ContainerStarted","Data":"47ec1fc59636c5edda596c6fea43a626b1e54fb2f09a136429af60a561be59c0"} Oct 31 17:42:10.311929 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:10.311894 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" event={"ID":"8ba87c0a-822f-467a-ab42-26cba2077357","Type":"ContainerStarted","Data":"51e05e0ebdaac5171f0efd757d0d95346aee692adbe5edddee75787a9b5d453a"} Oct 31 17:42:10.315458 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:10.314325 7618 generic.go:356] "Generic (PLEG): container finished" podID="e1ab7228-ad0c-4a75-b452-550e9db7bc8b" containerID="fb54c822ac096aaa50ccd3ca1d70e970c1493a8cd2d72edd1b827fdffc5d9dc1" exitCode=0 Oct 31 17:42:10.315458 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:10.314389 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" event={"ID":"e1ab7228-ad0c-4a75-b452-550e9db7bc8b","Type":"ContainerDied","Data":"fb54c822ac096aaa50ccd3ca1d70e970c1493a8cd2d72edd1b827fdffc5d9dc1"} Oct 31 17:42:10.357292 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:10.357190 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" podStartSLOduration=2.357169034 podStartE2EDuration="2.357169034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:42:10.344787758 +0000 UTC m=+852.092762242" watchObservedRunningTime="2026-10-31 17:42:10.357169034 +0000 UTC m=+852.105143502" Oct 31 17:42:10.802875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:10.802813 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" containerID="cri-o://23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0" gracePeriod=45 Oct 31 17:42:11.307705 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.307675 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:42:11.401168 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.401124 7618 generic.go:356] "Generic (PLEG): container finished" podID="8ba87c0a-822f-467a-ab42-26cba2077357" containerID="47ec1fc59636c5edda596c6fea43a626b1e54fb2f09a136429af60a561be59c0" exitCode=0 Oct 31 17:42:11.402397 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.401298 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" event={"ID":"8ba87c0a-822f-467a-ab42-26cba2077357","Type":"ContainerDied","Data":"47ec1fc59636c5edda596c6fea43a626b1e54fb2f09a136429af60a561be59c0"} Oct 31 17:42:11.409524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.403574 7618 generic.go:356] "Generic (PLEG): container finished" podID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerID="23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0" exitCode=0 Oct 31 17:42:11.409524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.403859 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" Oct 31 17:42:11.409524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.403977 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" event={"ID":"4e3c66c5-efb3-48a3-830b-bae3b90d74f5","Type":"ContainerDied","Data":"23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0"} Oct 31 17:42:11.409524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.404006 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk" event={"ID":"4e3c66c5-efb3-48a3-830b-bae3b90d74f5","Type":"ContainerDied","Data":"aba84e853f932c3d7f7e8cf82a6ea26f263e80da49b4e3ba24848e0cbca68e2c"} Oct 31 17:42:11.409524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.404028 7618 scope.go:122] "RemoveContainer" containerID="23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0" Oct 31 17:42:11.409524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.408392 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tls-certs\") pod \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " Oct 31 17:42:11.409524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.408489 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-home\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-home\") pod \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " Oct 31 17:42:11.409524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.408522 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tmp-dir\") pod \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " Oct 31 17:42:11.409524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.408658 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-model-cache\") pod \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " Oct 31 17:42:11.409524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.408831 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-kube-api-access-cpmlz\" (UniqueName: \"kubernetes.io/projected/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-kube-api-access-cpmlz\") pod \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " Oct 31 17:42:11.409524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.408899 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-dshm\") pod \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\" (UID: \"4e3c66c5-efb3-48a3-830b-bae3b90d74f5\") " Oct 31 17:42:11.410477 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.410219 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tmp-dir" pod "4e3c66c5-efb3-48a3-830b-bae3b90d74f5" (UID: "4e3c66c5-efb3-48a3-830b-bae3b90d74f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:11.413758 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.410811 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-model-cache" pod "4e3c66c5-efb3-48a3-830b-bae3b90d74f5" (UID: "4e3c66c5-efb3-48a3-830b-bae3b90d74f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:11.440523 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.437366 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-home" pod "4e3c66c5-efb3-48a3-830b-bae3b90d74f5" (UID: "4e3c66c5-efb3-48a3-830b-bae3b90d74f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:11.479562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.479459 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tls-certs" pod "4e3c66c5-efb3-48a3-830b-bae3b90d74f5" (UID: "4e3c66c5-efb3-48a3-830b-bae3b90d74f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:11.479897 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.479871 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-dshm" pod "4e3c66c5-efb3-48a3-830b-bae3b90d74f5" (UID: "4e3c66c5-efb3-48a3-830b-bae3b90d74f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:11.480037 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.479983 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-kube-api-access-cpmlz" pod "4e3c66c5-efb3-48a3-830b-bae3b90d74f5" (UID: "4e3c66c5-efb3-48a3-830b-bae3b90d74f5"). InnerVolumeSpecName "kube-api-access-cpmlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:11.496576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.490099 7618 scope.go:122] "RemoveContainer" containerID="23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0" Oct 31 17:42:11.496576 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:11.490726 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0\": container with ID starting with 23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0 not found: ID does not exist" containerID="23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0" Oct 31 17:42:11.496576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.490767 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0"} err="failed to get container status \"23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0\": rpc error: code = NotFound desc = could not find container \"23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0\": container with ID starting with 23314508561ab229db4274caa34f02c42a7b46a85767eeebafc2f5174f72eec0 not found: ID does not exist" Oct 31 17:42:11.514092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.513969 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpmlz\" (UniqueName: \"kubernetes.io/projected/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-kube-api-access-cpmlz\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:11.514092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.514009 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:11.514092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.514023 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:11.514092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.514036 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:11.514092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.514048 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:11.514092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.514060 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e3c66c5-efb3-48a3-830b-bae3b90d74f5-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:11.788316 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.787883 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx"] Oct 31 17:42:11.788654 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.788633 7618 cpu_manager.go:414] "containerMap: removing container" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" Oct 31 17:42:11.788755 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.788655 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" Oct 31 17:42:11.788755 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.788748 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" containerName="main" Oct 31 17:42:11.802112 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.802080 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk"] Oct 31 17:42:11.802399 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.802231 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:11.805762 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.804091 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5c446c/simulator-w2-5c446c-kserve-7b6f5cd548-wv9wk"] Oct 31 17:42:11.828981 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.828936 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx"] Oct 31 17:42:11.871720 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.871697 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:11.927645 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.927606 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thfc7\" (UniqueName: \"kubernetes.io/projected/8f95a8b4-8da7-4205-9e22-6d041871af15-kube-api-access-thfc7\") pod \"maas-api-revoke-sub-69af8050-vcrtx\" (UID: \"8f95a8b4-8da7-4205-9e22-6d041871af15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:11.927834 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.927713 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69af8050-vcrtx\" (UID: \"8f95a8b4-8da7-4205-9e22-6d041871af15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:11.927834 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:11.927779 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69af8050-vcrtx\" (UID: \"8f95a8b4-8da7-4205-9e22-6d041871af15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:12.029673 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.029580 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-openshift-ca\") pod \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\" (UID: \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\") " Oct 31 17:42:12.029673 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.029632 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-kube-api-access-pmbsk\" (UniqueName: \"kubernetes.io/projected/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-kube-api-access-pmbsk\") pod \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\" (UID: \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\") " Oct 31 17:42:12.029911 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.029839 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-xks-ca\") pod \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\" (UID: \"e1ab7228-ad0c-4a75-b452-550e9db7bc8b\") " Oct 31 17:42:12.030071 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.030051 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69af8050-vcrtx\" (UID: \"8f95a8b4-8da7-4205-9e22-6d041871af15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:12.030171 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.030115 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69af8050-vcrtx\" (UID: \"8f95a8b4-8da7-4205-9e22-6d041871af15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:12.030449 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.030413 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thfc7\" (UniqueName: \"kubernetes.io/projected/8f95a8b4-8da7-4205-9e22-6d041871af15-kube-api-access-thfc7\") pod \"maas-api-revoke-sub-69af8050-vcrtx\" (UID: \"8f95a8b4-8da7-4205-9e22-6d041871af15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:12.033050 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.031427 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-openshift-ca" pod "e1ab7228-ad0c-4a75-b452-550e9db7bc8b" (UID: "e1ab7228-ad0c-4a75-b452-550e9db7bc8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:12.033050 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.031993 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69af8050-vcrtx\" (UID: \"8f95a8b4-8da7-4205-9e22-6d041871af15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:12.035318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.035222 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-xks-ca" pod "e1ab7228-ad0c-4a75-b452-550e9db7bc8b" (UID: "e1ab7228-ad0c-4a75-b452-550e9db7bc8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:12.036646 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.036622 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69af8050-vcrtx\" (UID: \"8f95a8b4-8da7-4205-9e22-6d041871af15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:12.050181 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.050111 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thfc7\" (UniqueName: \"kubernetes.io/projected/8f95a8b4-8da7-4205-9e22-6d041871af15-kube-api-access-thfc7\") pod \"maas-api-revoke-sub-69af8050-vcrtx\" (UID: \"8f95a8b4-8da7-4205-9e22-6d041871af15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:12.057860 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.057801 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-kube-api-access-pmbsk" pod "e1ab7228-ad0c-4a75-b452-550e9db7bc8b" (UID: "e1ab7228-ad0c-4a75-b452-550e9db7bc8b"). InnerVolumeSpecName "kube-api-access-pmbsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:12.133972 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.133931 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:12.133972 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.133971 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmbsk\" (UniqueName: \"kubernetes.io/projected/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-kube-api-access-pmbsk\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:12.134184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.133987 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ab7228-ad0c-4a75-b452-550e9db7bc8b-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:12.170213 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.170177 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:12.377452 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.377369 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3c66c5-efb3-48a3-830b-bae3b90d74f5" path="/var/lib/kubelet/pods/4e3c66c5-efb3-48a3-830b-bae3b90d74f5/volumes" Oct 31 17:42:12.417905 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.416121 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" Oct 31 17:42:12.417905 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.416910 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2" event={"ID":"e1ab7228-ad0c-4a75-b452-550e9db7bc8b","Type":"ContainerDied","Data":"88cab59dcad1ccec0eafa7acd6a0aaf9ed77be9a7fc7c69816ccc27e1cd6d7d4"} Oct 31 17:42:12.417905 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.416960 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88cab59dcad1ccec0eafa7acd6a0aaf9ed77be9a7fc7c69816ccc27e1cd6d7d4" Oct 31 17:42:12.492285 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.492240 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx"] Oct 31 17:42:12.494160 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:42:12.494131 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f95a8b4_8da7_4205_9e22_6d041871af15.slice/crio-9319bbfc0b84fcaf43a87554f1497c921e2970d2dc1627396b428b0905f6705e WatchSource:0}: Error finding container 9319bbfc0b84fcaf43a87554f1497c921e2970d2dc1627396b428b0905f6705e: Status 404 returned error can't find the container with id 9319bbfc0b84fcaf43a87554f1497c921e2970d2dc1627396b428b0905f6705e Oct 31 17:42:12.580735 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.580703 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:12.749975 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.749936 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-xks-ca\") pod \"8ba87c0a-822f-467a-ab42-26cba2077357\" (UID: \"8ba87c0a-822f-467a-ab42-26cba2077357\") " Oct 31 17:42:12.750271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.750072 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-openshift-ca\") pod \"8ba87c0a-822f-467a-ab42-26cba2077357\" (UID: \"8ba87c0a-822f-467a-ab42-26cba2077357\") " Oct 31 17:42:12.750271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.750239 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba87c0a-822f-467a-ab42-26cba2077357-kube-api-access-6cr92\" (UniqueName: \"kubernetes.io/projected/8ba87c0a-822f-467a-ab42-26cba2077357-kube-api-access-6cr92\") pod \"8ba87c0a-822f-467a-ab42-26cba2077357\" (UID: \"8ba87c0a-822f-467a-ab42-26cba2077357\") " Oct 31 17:42:12.750576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.750526 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-openshift-ca" pod "8ba87c0a-822f-467a-ab42-26cba2077357" (UID: "8ba87c0a-822f-467a-ab42-26cba2077357"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:12.750999 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.750973 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:12.752886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.752756 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba87c0a-822f-467a-ab42-26cba2077357-kube-api-access-6cr92" pod "8ba87c0a-822f-467a-ab42-26cba2077357" (UID: "8ba87c0a-822f-467a-ab42-26cba2077357"). InnerVolumeSpecName "kube-api-access-6cr92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:12.753009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.752924 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-xks-ca" pod "8ba87c0a-822f-467a-ab42-26cba2077357" (UID: "8ba87c0a-822f-467a-ab42-26cba2077357"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:12.852289 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.852215 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba87c0a-822f-467a-ab42-26cba2077357-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:12.852289 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.852289 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cr92\" (UniqueName: \"kubernetes.io/projected/8ba87c0a-822f-467a-ab42-26cba2077357-kube-api-access-6cr92\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:12.895961 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.895921 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f"] Oct 31 17:42:12.901128 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:12.901096 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq"] Oct 31 17:42:13.440508 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:13.440478 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" Oct 31 17:42:13.440508 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:13.440488 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts" event={"ID":"8ba87c0a-822f-467a-ab42-26cba2077357","Type":"ContainerDied","Data":"51e05e0ebdaac5171f0efd757d0d95346aee692adbe5edddee75787a9b5d453a"} Oct 31 17:42:13.440782 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:13.440551 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e05e0ebdaac5171f0efd757d0d95346aee692adbe5edddee75787a9b5d453a" Oct 31 17:42:13.443699 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:13.443660 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" event={"ID":"8f95a8b4-8da7-4205-9e22-6d041871af15","Type":"ContainerStarted","Data":"e58c4c2e712fb4337a50d19451140cde7ac8e47b76dce64717f2bd1cd2f1e881"} Oct 31 17:42:13.443865 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:13.443721 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" event={"ID":"8f95a8b4-8da7-4205-9e22-6d041871af15","Type":"ContainerStarted","Data":"9319bbfc0b84fcaf43a87554f1497c921e2970d2dc1627396b428b0905f6705e"} Oct 31 17:42:13.479673 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:13.479612 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" podStartSLOduration=2.479596012 podStartE2EDuration="2.479596012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:42:13.476012682 +0000 UTC m=+855.223987162" watchObservedRunningTime="2026-10-31 17:42:13.479596012 +0000 UTC m=+855.227570545" Oct 31 17:42:14.464542 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.464504 7618 generic.go:356] "Generic (PLEG): container finished" podID="8f95a8b4-8da7-4205-9e22-6d041871af15" containerID="e58c4c2e712fb4337a50d19451140cde7ac8e47b76dce64717f2bd1cd2f1e881" exitCode=0 Oct 31 17:42:14.464986 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.464625 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" event={"ID":"8f95a8b4-8da7-4205-9e22-6d041871af15","Type":"ContainerDied","Data":"e58c4c2e712fb4337a50d19451140cde7ac8e47b76dce64717f2bd1cd2f1e881"} Oct 31 17:42:14.658356 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.658310 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz"] Oct 31 17:42:14.659024 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.659008 7618 cpu_manager.go:414] "containerMap: removing container" podUID="e1ab7228-ad0c-4a75-b452-550e9db7bc8b" containerName="revoke-keys" Oct 31 17:42:14.659064 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.659026 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ab7228-ad0c-4a75-b452-550e9db7bc8b" containerName="revoke-keys" Oct 31 17:42:14.659064 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.659035 7618 cpu_manager.go:414] "containerMap: removing container" podUID="8ba87c0a-822f-467a-ab42-26cba2077357" containerName="revoke-keys" Oct 31 17:42:14.659064 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.659040 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba87c0a-822f-467a-ab42-26cba2077357" containerName="revoke-keys" Oct 31 17:42:14.659181 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.659170 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ab7228-ad0c-4a75-b452-550e9db7bc8b" containerName="revoke-keys" Oct 31 17:42:14.659181 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.659182 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba87c0a-822f-467a-ab42-26cba2077357" containerName="revoke-keys" Oct 31 17:42:14.662562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.662540 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:14.682718 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.682682 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz"] Oct 31 17:42:14.782410 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.782315 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76kwf\" (UniqueName: \"kubernetes.io/projected/03356c26-fbaf-405d-9555-9fba69ce296f-kube-api-access-76kwf\") pod \"maas-api-revoke-sub-6201fc8b-wg7bz\" (UID: \"03356c26-fbaf-405d-9555-9fba69ce296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:14.782410 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.782392 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6201fc8b-wg7bz\" (UID: \"03356c26-fbaf-405d-9555-9fba69ce296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:14.782643 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.782548 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6201fc8b-wg7bz\" (UID: \"03356c26-fbaf-405d-9555-9fba69ce296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:14.883912 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.883868 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6201fc8b-wg7bz\" (UID: \"03356c26-fbaf-405d-9555-9fba69ce296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:14.884146 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.884130 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76kwf\" (UniqueName: \"kubernetes.io/projected/03356c26-fbaf-405d-9555-9fba69ce296f-kube-api-access-76kwf\") pod \"maas-api-revoke-sub-6201fc8b-wg7bz\" (UID: \"03356c26-fbaf-405d-9555-9fba69ce296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:14.884239 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.884209 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6201fc8b-wg7bz\" (UID: \"03356c26-fbaf-405d-9555-9fba69ce296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:14.884714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.884692 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6201fc8b-wg7bz\" (UID: \"03356c26-fbaf-405d-9555-9fba69ce296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:14.886867 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.886838 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6201fc8b-wg7bz\" (UID: \"03356c26-fbaf-405d-9555-9fba69ce296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:14.904742 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.904712 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76kwf\" (UniqueName: \"kubernetes.io/projected/03356c26-fbaf-405d-9555-9fba69ce296f-kube-api-access-76kwf\") pod \"maas-api-revoke-sub-6201fc8b-wg7bz\" (UID: \"03356c26-fbaf-405d-9555-9fba69ce296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:14.979421 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:14.979385 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:15.284735 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:15.284703 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz"] Oct 31 17:42:15.286692 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:42:15.286662 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03356c26_fbaf_405d_9555_9fba69ce296f.slice/crio-3f11a5b79341519c7189dda3a5c92622f76311f26c031fae1e2c275ccd0b73b8 WatchSource:0}: Error finding container 3f11a5b79341519c7189dda3a5c92622f76311f26c031fae1e2c275ccd0b73b8: Status 404 returned error can't find the container with id 3f11a5b79341519c7189dda3a5c92622f76311f26c031fae1e2c275ccd0b73b8 Oct 31 17:42:15.476638 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:15.476598 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" event={"ID":"03356c26-fbaf-405d-9555-9fba69ce296f","Type":"ContainerStarted","Data":"3f11a5b79341519c7189dda3a5c92622f76311f26c031fae1e2c275ccd0b73b8"} Oct 31 17:42:15.676951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:15.676921 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:15.701515 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:15.701475 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-openshift-ca\") pod \"8f95a8b4-8da7-4205-9e22-6d041871af15\" (UID: \"8f95a8b4-8da7-4205-9e22-6d041871af15\") " Oct 31 17:42:15.701713 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:15.701522 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f95a8b4-8da7-4205-9e22-6d041871af15-kube-api-access-thfc7\" (UniqueName: \"kubernetes.io/projected/8f95a8b4-8da7-4205-9e22-6d041871af15-kube-api-access-thfc7\") pod \"8f95a8b4-8da7-4205-9e22-6d041871af15\" (UID: \"8f95a8b4-8da7-4205-9e22-6d041871af15\") " Oct 31 17:42:15.701713 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:15.701578 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-xks-ca\") pod \"8f95a8b4-8da7-4205-9e22-6d041871af15\" (UID: \"8f95a8b4-8da7-4205-9e22-6d041871af15\") " Oct 31 17:42:15.701838 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:15.701797 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-openshift-ca" pod "8f95a8b4-8da7-4205-9e22-6d041871af15" (UID: "8f95a8b4-8da7-4205-9e22-6d041871af15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:15.702182 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:15.702163 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:15.703624 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:15.703602 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-xks-ca" pod "8f95a8b4-8da7-4205-9e22-6d041871af15" (UID: "8f95a8b4-8da7-4205-9e22-6d041871af15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:15.703758 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:15.703736 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f95a8b4-8da7-4205-9e22-6d041871af15-kube-api-access-thfc7" pod "8f95a8b4-8da7-4205-9e22-6d041871af15" (UID: "8f95a8b4-8da7-4205-9e22-6d041871af15"). InnerVolumeSpecName "kube-api-access-thfc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:15.804048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:15.804006 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thfc7\" (UniqueName: \"kubernetes.io/projected/8f95a8b4-8da7-4205-9e22-6d041871af15-kube-api-access-thfc7\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:15.804048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:15.804050 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f95a8b4-8da7-4205-9e22-6d041871af15-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:16.496903 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:16.496867 7618 generic.go:356] "Generic (PLEG): container finished" podID="03356c26-fbaf-405d-9555-9fba69ce296f" containerID="d794f304560ec837e9fe50166048d6a5bf2cd6aeeb3e791b00496b62001be5fb" exitCode=0 Oct 31 17:42:16.497424 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:16.496945 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" event={"ID":"03356c26-fbaf-405d-9555-9fba69ce296f","Type":"ContainerDied","Data":"d794f304560ec837e9fe50166048d6a5bf2cd6aeeb3e791b00496b62001be5fb"} Oct 31 17:42:16.498556 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:16.498531 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" Oct 31 17:42:16.498556 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:16.498547 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx" event={"ID":"8f95a8b4-8da7-4205-9e22-6d041871af15","Type":"ContainerDied","Data":"9319bbfc0b84fcaf43a87554f1497c921e2970d2dc1627396b428b0905f6705e"} Oct 31 17:42:16.498711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:16.498595 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9319bbfc0b84fcaf43a87554f1497c921e2970d2dc1627396b428b0905f6705e" Oct 31 17:42:17.698952 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:17.698925 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:17.738937 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:17.738904 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03356c26-fbaf-405d-9555-9fba69ce296f-kube-api-access-76kwf\" (UniqueName: \"kubernetes.io/projected/03356c26-fbaf-405d-9555-9fba69ce296f-kube-api-access-76kwf\") pod \"03356c26-fbaf-405d-9555-9fba69ce296f\" (UID: \"03356c26-fbaf-405d-9555-9fba69ce296f\") " Oct 31 17:42:17.739156 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:17.739142 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-openshift-ca\") pod \"03356c26-fbaf-405d-9555-9fba69ce296f\" (UID: \"03356c26-fbaf-405d-9555-9fba69ce296f\") " Oct 31 17:42:17.739211 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:17.739169 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-xks-ca\") pod \"03356c26-fbaf-405d-9555-9fba69ce296f\" (UID: \"03356c26-fbaf-405d-9555-9fba69ce296f\") " Oct 31 17:42:17.739723 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:17.739551 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-openshift-ca" pod "03356c26-fbaf-405d-9555-9fba69ce296f" (UID: "03356c26-fbaf-405d-9555-9fba69ce296f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:17.739926 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:17.739843 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:17.741324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:17.741296 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03356c26-fbaf-405d-9555-9fba69ce296f-kube-api-access-76kwf" pod "03356c26-fbaf-405d-9555-9fba69ce296f" (UID: "03356c26-fbaf-405d-9555-9fba69ce296f"). InnerVolumeSpecName "kube-api-access-76kwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:17.742143 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:17.742112 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-xks-ca" pod "03356c26-fbaf-405d-9555-9fba69ce296f" (UID: "03356c26-fbaf-405d-9555-9fba69ce296f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:17.840991 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:17.840909 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03356c26-fbaf-405d-9555-9fba69ce296f-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:17.840991 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:17.840938 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76kwf\" (UniqueName: \"kubernetes.io/projected/03356c26-fbaf-405d-9555-9fba69ce296f-kube-api-access-76kwf\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:18.270827 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.270782 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67"] Oct 31 17:42:18.271552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.271531 7618 cpu_manager.go:414] "containerMap: removing container" podUID="8f95a8b4-8da7-4205-9e22-6d041871af15" containerName="revoke-keys" Oct 31 17:42:18.271653 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.271551 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f95a8b4-8da7-4205-9e22-6d041871af15" containerName="revoke-keys" Oct 31 17:42:18.271653 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.271579 7618 cpu_manager.go:414] "containerMap: removing container" podUID="03356c26-fbaf-405d-9555-9fba69ce296f" containerName="revoke-keys" Oct 31 17:42:18.271653 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.271586 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03356c26-fbaf-405d-9555-9fba69ce296f" containerName="revoke-keys" Oct 31 17:42:18.271819 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.271810 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f95a8b4-8da7-4205-9e22-6d041871af15" containerName="revoke-keys" Oct 31 17:42:18.271870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.271833 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="03356c26-fbaf-405d-9555-9fba69ce296f" containerName="revoke-keys" Oct 31 17:42:18.275885 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.275861 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:18.293932 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.293895 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67"] Oct 31 17:42:18.348678 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.348608 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a1b7134-snq67\" (UID: \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:18.348858 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.348785 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a1b7134-snq67\" (UID: \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:18.348932 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.348919 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9k6\" (UniqueName: \"kubernetes.io/projected/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-kube-api-access-hb9k6\") pod \"maas-api-revoke-sub-0a1b7134-snq67\" (UID: \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:18.450577 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.450532 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a1b7134-snq67\" (UID: \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:18.450784 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.450674 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a1b7134-snq67\" (UID: \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:18.450930 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.450819 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9k6\" (UniqueName: \"kubernetes.io/projected/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-kube-api-access-hb9k6\") pod \"maas-api-revoke-sub-0a1b7134-snq67\" (UID: \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:18.451492 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.451466 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a1b7134-snq67\" (UID: \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:18.453345 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.453318 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a1b7134-snq67\" (UID: \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:18.473336 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.473302 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9k6\" (UniqueName: \"kubernetes.io/projected/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-kube-api-access-hb9k6\") pod \"maas-api-revoke-sub-0a1b7134-snq67\" (UID: \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:18.525353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.525214 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" Oct 31 17:42:18.525520 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.525480 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz" event={"ID":"03356c26-fbaf-405d-9555-9fba69ce296f","Type":"ContainerDied","Data":"3f11a5b79341519c7189dda3a5c92622f76311f26c031fae1e2c275ccd0b73b8"} Oct 31 17:42:18.525575 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.525537 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f11a5b79341519c7189dda3a5c92622f76311f26c031fae1e2c275ccd0b73b8" Oct 31 17:42:18.593950 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.593913 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:18.688822 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.688782 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr"] Oct 31 17:42:18.902360 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:18.902333 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67"] Oct 31 17:42:18.903081 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:42:18.903052 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84d61c3_fa1d_46dc_9e72_1e5f58d05ff7.slice/crio-3f3ceaa7e65bd2507abfc932e50cabc7ba9b0cf1e5311a73e832ecc98a1afb12 WatchSource:0}: Error finding container 3f3ceaa7e65bd2507abfc932e50cabc7ba9b0cf1e5311a73e832ecc98a1afb12: Status 404 returned error can't find the container with id 3f3ceaa7e65bd2507abfc932e50cabc7ba9b0cf1e5311a73e832ecc98a1afb12 Oct 31 17:42:19.265435 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.265392 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" containerID="cri-o://5f678654ede31354060cb3baf696c6c4bc4a8ab23ef2fbadbd4bd3d49eff676d" gracePeriod=45 Oct 31 17:42:19.296361 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.296325 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" containerID="cri-o://03469e47bea4bbcd68ca439038b9808122d4ef0ac93e1634cc0f931828a80565" gracePeriod=44 Oct 31 17:42:19.543697 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.543440 7618 generic.go:356] "Generic (PLEG): container finished" podID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerID="5f678654ede31354060cb3baf696c6c4bc4a8ab23ef2fbadbd4bd3d49eff676d" exitCode=0 Oct 31 17:42:19.543697 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.543599 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" event={"ID":"cacd0841-f34a-4ad3-8534-e13306ad2037","Type":"ContainerDied","Data":"5f678654ede31354060cb3baf696c6c4bc4a8ab23ef2fbadbd4bd3d49eff676d"} Oct 31 17:42:19.545946 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.545847 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" event={"ID":"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7","Type":"ContainerStarted","Data":"edd4ebf31beaeacb6d5ef3f8cd3f20556e351062697fdaf001d6c042f8cae617"} Oct 31 17:42:19.545946 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.545913 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" event={"ID":"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7","Type":"ContainerStarted","Data":"3f3ceaa7e65bd2507abfc932e50cabc7ba9b0cf1e5311a73e832ecc98a1afb12"} Oct 31 17:42:19.548372 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.548347 7618 generic.go:356] "Generic (PLEG): container finished" podID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerID="03469e47bea4bbcd68ca439038b9808122d4ef0ac93e1634cc0f931828a80565" exitCode=0 Oct 31 17:42:19.548498 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.548422 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" event={"ID":"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5","Type":"ContainerDied","Data":"03469e47bea4bbcd68ca439038b9808122d4ef0ac93e1634cc0f931828a80565"} Oct 31 17:42:19.582675 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.580448 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" podStartSLOduration=1.580429495 podStartE2EDuration="1.580429495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:42:19.579781811 +0000 UTC m=+861.327756279" watchObservedRunningTime="2026-10-31 17:42:19.580429495 +0000 UTC m=+861.328403961" Oct 31 17:42:19.830772 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.830740 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:42:19.832850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.832822 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:42:19.892320 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.892289 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-dshm\") pod \"cacd0841-f34a-4ad3-8534-e13306ad2037\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " Oct 31 17:42:19.892511 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.892349 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cacd0841-f34a-4ad3-8534-e13306ad2037-tls-certs\" (UniqueName: \"kubernetes.io/secret/cacd0841-f34a-4ad3-8534-e13306ad2037-tls-certs\") pod \"cacd0841-f34a-4ad3-8534-e13306ad2037\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " Oct 31 17:42:19.892511 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.892380 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-home\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-home\") pod \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " Oct 31 17:42:19.892511 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.892431 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tmp-dir\") pod \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " Oct 31 17:42:19.892691 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.892537 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-tmp-dir\") pod \"cacd0841-f34a-4ad3-8534-e13306ad2037\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " Oct 31 17:42:19.892691 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.892587 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cacd0841-f34a-4ad3-8534-e13306ad2037-kube-api-access-glhdn\" (UniqueName: \"kubernetes.io/projected/cacd0841-f34a-4ad3-8534-e13306ad2037-kube-api-access-glhdn\") pod \"cacd0841-f34a-4ad3-8534-e13306ad2037\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " Oct 31 17:42:19.892691 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.892652 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-kube-api-access-7sbfz\" (UniqueName: \"kubernetes.io/projected/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-kube-api-access-7sbfz\") pod \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " Oct 31 17:42:19.892691 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.892656 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-home" pod "05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" (UID: "05ee47cc-590f-4de3-ab6d-0ce5d97c41d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:19.892894 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.892758 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tls-certs\") pod \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " Oct 31 17:42:19.892894 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.892827 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-model-cache\") pod \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " Oct 31 17:42:19.892989 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.892915 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tmp-dir" pod "05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" (UID: "05ee47cc-590f-4de3-ab6d-0ce5d97c41d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:19.892989 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.892935 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-tmp-dir" pod "cacd0841-f34a-4ad3-8534-e13306ad2037" (UID: "cacd0841-f34a-4ad3-8534-e13306ad2037"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:19.893320 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.893295 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-home\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-home\") pod \"cacd0841-f34a-4ad3-8534-e13306ad2037\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " Oct 31 17:42:19.893443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.893428 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-model-cache" pod "05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" (UID: "05ee47cc-590f-4de3-ab6d-0ce5d97c41d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:19.893573 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.893544 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-home" pod "cacd0841-f34a-4ad3-8534-e13306ad2037" (UID: "cacd0841-f34a-4ad3-8534-e13306ad2037"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:19.893744 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.893728 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-model-cache\") pod \"cacd0841-f34a-4ad3-8534-e13306ad2037\" (UID: \"cacd0841-f34a-4ad3-8534-e13306ad2037\") " Oct 31 17:42:19.893889 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.893854 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-dshm\") pod \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\" (UID: \"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5\") " Oct 31 17:42:19.894882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.894127 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-model-cache" pod "cacd0841-f34a-4ad3-8534-e13306ad2037" (UID: "cacd0841-f34a-4ad3-8534-e13306ad2037"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:19.894882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.894771 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:19.894882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.894795 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:19.894882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.894819 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:19.894882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.894831 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:19.894882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.894845 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:19.894882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.894859 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:19.896097 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.896064 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tls-certs" pod "05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" (UID: "05ee47cc-590f-4de3-ab6d-0ce5d97c41d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:19.896097 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.896089 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-kube-api-access-7sbfz" pod "05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" (UID: "05ee47cc-590f-4de3-ab6d-0ce5d97c41d5"). InnerVolumeSpecName "kube-api-access-7sbfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:19.898576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.898551 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-dshm" pod "05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" (UID: "05ee47cc-590f-4de3-ab6d-0ce5d97c41d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:19.898576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.898563 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cacd0841-f34a-4ad3-8534-e13306ad2037-tls-certs" pod "cacd0841-f34a-4ad3-8534-e13306ad2037" (UID: "cacd0841-f34a-4ad3-8534-e13306ad2037"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:19.898702 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.898564 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacd0841-f34a-4ad3-8534-e13306ad2037-kube-api-access-glhdn" pod "cacd0841-f34a-4ad3-8534-e13306ad2037" (UID: "cacd0841-f34a-4ad3-8534-e13306ad2037"). InnerVolumeSpecName "kube-api-access-glhdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:19.899928 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.899907 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-dshm" pod "cacd0841-f34a-4ad3-8534-e13306ad2037" (UID: "cacd0841-f34a-4ad3-8534-e13306ad2037"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:19.996097 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.996055 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sbfz\" (UniqueName: \"kubernetes.io/projected/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-kube-api-access-7sbfz\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:19.996097 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.996089 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:19.996097 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.996101 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:19.996612 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.996109 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cacd0841-f34a-4ad3-8534-e13306ad2037-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:19.996612 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.996119 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cacd0841-f34a-4ad3-8534-e13306ad2037-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:19.996612 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:19.996126 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhdn\" (UniqueName: \"kubernetes.io/projected/cacd0841-f34a-4ad3-8534-e13306ad2037-kube-api-access-glhdn\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:20.138635 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.138588 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" containerID="cri-o://91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137" gracePeriod=44 Oct 31 17:42:20.469522 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.469482 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:42:20.506626 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.506593 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-dshm\") pod \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " Oct 31 17:42:20.506832 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.506650 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-model-cache\") pod \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " Oct 31 17:42:20.506832 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.506722 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tmp-dir\") pod \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " Oct 31 17:42:20.506832 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.506777 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-kube-api-access-fn8vs\" (UniqueName: \"kubernetes.io/projected/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-kube-api-access-fn8vs\") pod \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " Oct 31 17:42:20.506832 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.506829 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-home\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-home\") pod \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " Oct 31 17:42:20.507177 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.506861 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tls-certs\") pod \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\" (UID: \"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6\") " Oct 31 17:42:20.507938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.507912 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tmp-dir" pod "fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" (UID: "fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:20.511877 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.511702 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-model-cache" pod "fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" (UID: "fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:20.512078 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.512042 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-home" pod "fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" (UID: "fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:20.513901 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.513876 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-dshm" pod "fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" (UID: "fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:20.525389 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.525345 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tls-certs" pod "fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" (UID: "fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:20.529434 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.529377 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-kube-api-access-fn8vs" pod "fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" (UID: "fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6"). InnerVolumeSpecName "kube-api-access-fn8vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:20.560529 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.560473 7618 generic.go:356] "Generic (PLEG): container finished" podID="e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7" containerID="edd4ebf31beaeacb6d5ef3f8cd3f20556e351062697fdaf001d6c042f8cae617" exitCode=0 Oct 31 17:42:20.560724 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.560606 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" event={"ID":"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7","Type":"ContainerDied","Data":"edd4ebf31beaeacb6d5ef3f8cd3f20556e351062697fdaf001d6c042f8cae617"} Oct 31 17:42:20.562757 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.562627 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" event={"ID":"05ee47cc-590f-4de3-ab6d-0ce5d97c41d5","Type":"ContainerDied","Data":"270e8553ec8fe25c5f3c053486c223ddb7830fe79e94bc5195373ac55497ff34"} Oct 31 17:42:20.562757 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.562670 7618 scope.go:122] "RemoveContainer" containerID="03469e47bea4bbcd68ca439038b9808122d4ef0ac93e1634cc0f931828a80565" Oct 31 17:42:20.562967 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.562822 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" Oct 31 17:42:20.566437 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.566401 7618 generic.go:356] "Generic (PLEG): container finished" podID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerID="91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137" exitCode=0 Oct 31 17:42:20.566666 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.566642 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" event={"ID":"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6","Type":"ContainerDied","Data":"91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137"} Oct 31 17:42:20.566745 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.566685 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" event={"ID":"fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6","Type":"ContainerDied","Data":"2b831e8d32f72d28cad1665e1cb849572cf8898654966316d7f74d315659ab79"} Oct 31 17:42:20.566817 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.566747 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5" Oct 31 17:42:20.573490 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.573451 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" event={"ID":"cacd0841-f34a-4ad3-8534-e13306ad2037","Type":"ContainerDied","Data":"349b6c73406ce6e063afe466872ff9ad02c41130ba831600cdb7b5b7c2f444b8"} Oct 31 17:42:20.573619 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.573557 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8" Oct 31 17:42:20.588068 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.588046 7618 scope.go:122] "RemoveContainer" containerID="91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137" Oct 31 17:42:20.601006 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.600979 7618 scope.go:122] "RemoveContainer" containerID="91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137" Oct 31 17:42:20.601395 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:20.601366 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137\": container with ID starting with 91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137 not found: ID does not exist" containerID="91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137" Oct 31 17:42:20.601499 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.601404 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137"} err="failed to get container status \"91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137\": rpc error: code = NotFound desc = could not find container \"91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137\": container with ID starting with 91f2e0722df2febeded5f333ee30c135bcc59acb4855dacc1913045b95e78137 not found: ID does not exist" Oct 31 17:42:20.601499 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.601425 7618 scope.go:122] "RemoveContainer" containerID="5f678654ede31354060cb3baf696c6c4bc4a8ab23ef2fbadbd4bd3d49eff676d" Oct 31 17:42:20.609409 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.609370 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:20.609409 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.609405 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn8vs\" (UniqueName: \"kubernetes.io/projected/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-kube-api-access-fn8vs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:20.609611 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.609420 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:20.609611 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.609434 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:20.609611 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.609447 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:20.609611 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.609459 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:20.633373 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.633337 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz"] Oct 31 17:42:20.636008 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.635980 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz"] Oct 31 17:42:20.663440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.663306 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8"] Oct 31 17:42:20.670935 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.670900 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/distinct-w3-cf0a86-kserve-d764fddbb-nczg8"] Oct 31 17:42:20.699353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.699315 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5"] Oct 31 17:42:20.705554 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.705522 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/simulator-w3-cf0a86-kserve-549d995b6b-xzql5"] Oct 31 17:42:20.799512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:20.799474 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/distinct-2-w3-cf0a86-kserve-649db8fdb9-8djpz" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:21.464693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.464653 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4"] Oct 31 17:42:21.465363 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.465343 7618 cpu_manager.go:414] "containerMap: removing container" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" Oct 31 17:42:21.465443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.465364 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" Oct 31 17:42:21.465443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.465395 7618 cpu_manager.go:414] "containerMap: removing container" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" Oct 31 17:42:21.465443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.465402 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" Oct 31 17:42:21.465443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.465426 7618 cpu_manager.go:414] "containerMap: removing container" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" Oct 31 17:42:21.465443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.465433 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" Oct 31 17:42:21.465596 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.465542 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" containerName="main" Oct 31 17:42:21.465596 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.465559 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" containerName="main" Oct 31 17:42:21.465596 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.465568 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" containerName="main" Oct 31 17:42:21.470839 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.470807 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:21.489711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.489675 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4"] Oct 31 17:42:21.524503 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.524468 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkxgw\" (UniqueName: \"kubernetes.io/projected/54de558b-f489-46a4-8978-de66c15583d7-kube-api-access-hkxgw\") pod \"maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4\" (UID: \"54de558b-f489-46a4-8978-de66c15583d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:21.524728 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.524549 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de558b-f489-46a4-8978-de66c15583d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4\" (UID: \"54de558b-f489-46a4-8978-de66c15583d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:21.524798 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.524724 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de558b-f489-46a4-8978-de66c15583d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4\" (UID: \"54de558b-f489-46a4-8978-de66c15583d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:21.626876 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.626831 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkxgw\" (UniqueName: \"kubernetes.io/projected/54de558b-f489-46a4-8978-de66c15583d7-kube-api-access-hkxgw\") pod \"maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4\" (UID: \"54de558b-f489-46a4-8978-de66c15583d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:21.627077 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.626937 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de558b-f489-46a4-8978-de66c15583d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4\" (UID: \"54de558b-f489-46a4-8978-de66c15583d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:21.627077 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.627052 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de558b-f489-46a4-8978-de66c15583d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4\" (UID: \"54de558b-f489-46a4-8978-de66c15583d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:21.627827 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.627802 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de558b-f489-46a4-8978-de66c15583d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4\" (UID: \"54de558b-f489-46a4-8978-de66c15583d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:21.629934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.629908 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de558b-f489-46a4-8978-de66c15583d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4\" (UID: \"54de558b-f489-46a4-8978-de66c15583d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:21.649298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.645500 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkxgw\" (UniqueName: \"kubernetes.io/projected/54de558b-f489-46a4-8978-de66c15583d7-kube-api-access-hkxgw\") pod \"maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4\" (UID: \"54de558b-f489-46a4-8978-de66c15583d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:21.782929 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.782897 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:21.786653 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.786625 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:21.834294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.831664 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-xks-ca\") pod \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\" (UID: \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\") " Oct 31 17:42:21.834294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.831850 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-openshift-ca\") pod \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\" (UID: \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\") " Oct 31 17:42:21.834294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.832226 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-kube-api-access-hb9k6\" (UniqueName: \"kubernetes.io/projected/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-kube-api-access-hb9k6\") pod \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\" (UID: \"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7\") " Oct 31 17:42:21.834294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.833414 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-openshift-ca" pod "e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7" (UID: "e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:21.838577 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.838013 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-kube-api-access-hb9k6" pod "e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7" (UID: "e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7"). InnerVolumeSpecName "kube-api-access-hb9k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:21.846933 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.846881 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-xks-ca" pod "e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7" (UID: "e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:21.933984 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.933952 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:21.934179 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.934167 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:21.934234 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:21.934227 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb9k6\" (UniqueName: \"kubernetes.io/projected/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7-kube-api-access-hb9k6\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:22.122940 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.122907 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4"] Oct 31 17:42:22.123480 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:42:22.123455 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54de558b_f489_46a4_8978_de66c15583d7.slice/crio-6a8f566721ad19f464431ba869b4639f3d27cf6445504937bcd783e423a4ef55 WatchSource:0}: Error finding container 6a8f566721ad19f464431ba869b4639f3d27cf6445504937bcd783e423a4ef55: Status 404 returned error can't find the container with id 6a8f566721ad19f464431ba869b4639f3d27cf6445504937bcd783e423a4ef55 Oct 31 17:42:22.245308 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.241566 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" containerID="cri-o://9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0" gracePeriod=42 Oct 31 17:42:22.298528 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.298487 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused" Oct 31 17:42:22.375995 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.375921 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ee47cc-590f-4de3-ab6d-0ce5d97c41d5" path="/var/lib/kubelet/pods/05ee47cc-590f-4de3-ab6d-0ce5d97c41d5/volumes" Oct 31 17:42:22.376465 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.376450 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cacd0841-f34a-4ad3-8534-e13306ad2037" path="/var/lib/kubelet/pods/cacd0841-f34a-4ad3-8534-e13306ad2037/volumes" Oct 31 17:42:22.376852 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.376840 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6" path="/var/lib/kubelet/pods/fb7376f2-8c9a-4f09-89f8-fd8ee88e80c6/volumes" Oct 31 17:42:22.563318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.563283 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:42:22.598699 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.598652 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" event={"ID":"54de558b-f489-46a4-8978-de66c15583d7","Type":"ContainerStarted","Data":"fcd672852127230b9d808f460265d1b8cfa9d925c320cdd4c0e748523484667d"} Oct 31 17:42:22.598940 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.598771 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" event={"ID":"54de558b-f489-46a4-8978-de66c15583d7","Type":"ContainerStarted","Data":"6a8f566721ad19f464431ba869b4639f3d27cf6445504937bcd783e423a4ef55"} Oct 31 17:42:22.601293 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.601269 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" Oct 31 17:42:22.601720 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.601683 7618 generic.go:356] "Generic (PLEG): container finished" podID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerID="9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0" exitCode=0 Oct 31 17:42:22.601845 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.601818 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" event={"ID":"d3b13cb1-a3f6-4421-b295-547e34fd90be","Type":"ContainerDied","Data":"9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0"} Oct 31 17:42:22.601907 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.601849 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96" event={"ID":"d3b13cb1-a3f6-4421-b295-547e34fd90be","Type":"ContainerDied","Data":"43afb1b50300aa8377716df219e7e2b42c87737824095ecb1f56eaf29b75bf5e"} Oct 31 17:42:22.601907 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.601870 7618 scope.go:122] "RemoveContainer" containerID="9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0" Oct 31 17:42:22.605068 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.605045 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" event={"ID":"e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7","Type":"ContainerDied","Data":"3f3ceaa7e65bd2507abfc932e50cabc7ba9b0cf1e5311a73e832ecc98a1afb12"} Oct 31 17:42:22.605345 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.605239 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f3ceaa7e65bd2507abfc932e50cabc7ba9b0cf1e5311a73e832ecc98a1afb12" Oct 31 17:42:22.605503 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.605491 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67" Oct 31 17:42:22.629126 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.629055 7618 scope.go:122] "RemoveContainer" containerID="9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0" Oct 31 17:42:22.631747 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:22.631716 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0\": container with ID starting with 9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0 not found: ID does not exist" containerID="9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0" Oct 31 17:42:22.632064 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.632001 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0"} err="failed to get container status \"9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0\": rpc error: code = NotFound desc = could not find container \"9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0\": container with ID starting with 9dcc4664f99ed77f6197cb7cb9c6adf1ec7944a3e2c1c92168829c442c8444f0 not found: ID does not exist" Oct 31 17:42:22.652833 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.652786 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-model-cache\") pod \"d3b13cb1-a3f6-4421-b295-547e34fd90be\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " Oct 31 17:42:22.653036 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.652839 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b13cb1-a3f6-4421-b295-547e34fd90be-kube-api-access-54q2s\" (UniqueName: \"kubernetes.io/projected/d3b13cb1-a3f6-4421-b295-547e34fd90be-kube-api-access-54q2s\") pod \"d3b13cb1-a3f6-4421-b295-547e34fd90be\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " Oct 31 17:42:22.653036 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.652880 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3b13cb1-a3f6-4421-b295-547e34fd90be-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b13cb1-a3f6-4421-b295-547e34fd90be-tls-certs\") pod \"d3b13cb1-a3f6-4421-b295-547e34fd90be\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " Oct 31 17:42:22.653036 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.652925 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-tmp-dir\") pod \"d3b13cb1-a3f6-4421-b295-547e34fd90be\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " Oct 31 17:42:22.653218 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.652957 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-home\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-home\") pod \"d3b13cb1-a3f6-4421-b295-547e34fd90be\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " Oct 31 17:42:22.653218 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.653092 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-dshm\") pod \"d3b13cb1-a3f6-4421-b295-547e34fd90be\" (UID: \"d3b13cb1-a3f6-4421-b295-547e34fd90be\") " Oct 31 17:42:22.653585 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.653561 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-tmp-dir" pod "d3b13cb1-a3f6-4421-b295-547e34fd90be" (UID: "d3b13cb1-a3f6-4421-b295-547e34fd90be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:22.653774 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.653751 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-home" pod "d3b13cb1-a3f6-4421-b295-547e34fd90be" (UID: "d3b13cb1-a3f6-4421-b295-547e34fd90be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:22.654220 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.654188 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-model-cache" pod "d3b13cb1-a3f6-4421-b295-547e34fd90be" (UID: "d3b13cb1-a3f6-4421-b295-547e34fd90be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:22.654559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.654539 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:22.654669 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.654656 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:22.654768 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.654756 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:22.656736 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.656137 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b13cb1-a3f6-4421-b295-547e34fd90be-tls-certs" pod "d3b13cb1-a3f6-4421-b295-547e34fd90be" (UID: "d3b13cb1-a3f6-4421-b295-547e34fd90be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:22.656970 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.656937 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b13cb1-a3f6-4421-b295-547e34fd90be-kube-api-access-54q2s" pod "d3b13cb1-a3f6-4421-b295-547e34fd90be" (UID: "d3b13cb1-a3f6-4421-b295-547e34fd90be"). InnerVolumeSpecName "kube-api-access-54q2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:22.657961 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.657918 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" podStartSLOduration=1.657906243 podStartE2EDuration="1.657906243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:42:22.634978121 +0000 UTC m=+864.382952590" watchObservedRunningTime="2026-10-31 17:42:22.657906243 +0000 UTC m=+864.405880764" Oct 31 17:42:22.665324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.660428 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-dshm" pod "d3b13cb1-a3f6-4421-b295-547e34fd90be" (UID: "d3b13cb1-a3f6-4421-b295-547e34fd90be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:22.756744 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.756703 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54q2s\" (UniqueName: \"kubernetes.io/projected/d3b13cb1-a3f6-4421-b295-547e34fd90be-kube-api-access-54q2s\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:22.756744 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.756742 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b13cb1-a3f6-4421-b295-547e34fd90be-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:22.756954 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.756756 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b13cb1-a3f6-4421-b295-547e34fd90be-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:22.948530 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.948493 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96"] Oct 31 17:42:22.953084 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:22.953048 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cf0a86/premium-w3-cf0a86-kserve-795967bb4c-jrw96"] Oct 31 17:42:23.617654 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:23.617617 7618 generic.go:356] "Generic (PLEG): container finished" podID="54de558b-f489-46a4-8978-de66c15583d7" containerID="fcd672852127230b9d808f460265d1b8cfa9d925c320cdd4c0e748523484667d" exitCode=0 Oct 31 17:42:23.618241 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:23.617694 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" event={"ID":"54de558b-f489-46a4-8978-de66c15583d7","Type":"ContainerDied","Data":"fcd672852127230b9d808f460265d1b8cfa9d925c320cdd4c0e748523484667d"} Oct 31 17:42:23.689455 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:23.689408 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" podUID="8c01da1b-48fc-49e4-9341-54965a5c45a0" containerName="payload-processing" containerID="cri-o://878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767" gracePeriod=25 Oct 31 17:42:24.376020 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:24.375988 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" path="/var/lib/kubelet/pods/d3b13cb1-a3f6-4421-b295-547e34fd90be/volumes" Oct 31 17:42:24.410715 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:24.410675 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5"] Oct 31 17:42:24.801755 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:24.801727 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:24.889853 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:24.889823 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54de558b-f489-46a4-8978-de66c15583d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de558b-f489-46a4-8978-de66c15583d7-maas-api-openshift-ca\") pod \"54de558b-f489-46a4-8978-de66c15583d7\" (UID: \"54de558b-f489-46a4-8978-de66c15583d7\") " Oct 31 17:42:24.890008 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:24.889928 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54de558b-f489-46a4-8978-de66c15583d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de558b-f489-46a4-8978-de66c15583d7-maas-api-xks-ca\") pod \"54de558b-f489-46a4-8978-de66c15583d7\" (UID: \"54de558b-f489-46a4-8978-de66c15583d7\") " Oct 31 17:42:24.890008 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:24.889977 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54de558b-f489-46a4-8978-de66c15583d7-kube-api-access-hkxgw\" (UniqueName: \"kubernetes.io/projected/54de558b-f489-46a4-8978-de66c15583d7-kube-api-access-hkxgw\") pod \"54de558b-f489-46a4-8978-de66c15583d7\" (UID: \"54de558b-f489-46a4-8978-de66c15583d7\") " Oct 31 17:42:24.890545 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:24.890436 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54de558b-f489-46a4-8978-de66c15583d7-maas-api-openshift-ca" pod "54de558b-f489-46a4-8978-de66c15583d7" (UID: "54de558b-f489-46a4-8978-de66c15583d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:24.891055 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:24.891029 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de558b-f489-46a4-8978-de66c15583d7-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:24.892843 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:24.892816 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54de558b-f489-46a4-8978-de66c15583d7-maas-api-xks-ca" pod "54de558b-f489-46a4-8978-de66c15583d7" (UID: "54de558b-f489-46a4-8978-de66c15583d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:24.892942 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:24.892870 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54de558b-f489-46a4-8978-de66c15583d7-kube-api-access-hkxgw" pod "54de558b-f489-46a4-8978-de66c15583d7" (UID: "54de558b-f489-46a4-8978-de66c15583d7"). InnerVolumeSpecName "kube-api-access-hkxgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:24.992880 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:24.992835 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de558b-f489-46a4-8978-de66c15583d7-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:24.992880 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:24.992871 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkxgw\" (UniqueName: \"kubernetes.io/projected/54de558b-f489-46a4-8978-de66c15583d7-kube-api-access-hkxgw\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:25.640826 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:25.640795 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" Oct 31 17:42:25.641015 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:25.640794 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4" event={"ID":"54de558b-f489-46a4-8978-de66c15583d7","Type":"ContainerDied","Data":"6a8f566721ad19f464431ba869b4639f3d27cf6445504937bcd783e423a4ef55"} Oct 31 17:42:25.641015 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:25.640985 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a8f566721ad19f464431ba869b4639f3d27cf6445504937bcd783e423a4ef55" Oct 31 17:42:26.235777 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:26.235721 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl"] Oct 31 17:42:27.920341 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:27.920295 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" containerID="cri-o://f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940" gracePeriod=45 Oct 31 17:42:27.923118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:27.923055 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" containerID="cri-o://8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4" gracePeriod=45 Oct 31 17:42:28.067542 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.067500 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t"] Oct 31 17:42:28.101464 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.101425 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": dial tcp 10.134.0.59:8000: connect: connection refused" Oct 31 17:42:28.154276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.153855 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" podUID="8c01da1b-48fc-49e4-9341-54965a5c45a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 17:42:28.303020 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.302979 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:42:28.309677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.309643 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:42:28.473063 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.473022 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b962c13d-7616-4971-9ae1-5b36c812e296-kube-api-access-jhsc7\" (UniqueName: \"kubernetes.io/projected/b962c13d-7616-4971-9ae1-5b36c812e296-kube-api-access-jhsc7\") pod \"b962c13d-7616-4971-9ae1-5b36c812e296\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " Oct 31 17:42:28.473236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.473076 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-dshm\") pod \"baeaac10-b796-4dcd-9d24-752c346dff6a\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " Oct 31 17:42:28.473236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.473133 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-tmp-dir\") pod \"b962c13d-7616-4971-9ae1-5b36c812e296\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " Oct 31 17:42:28.473236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.473193 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-home\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-home\") pod \"baeaac10-b796-4dcd-9d24-752c346dff6a\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " Oct 31 17:42:28.473236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.473230 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-home\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-home\") pod \"b962c13d-7616-4971-9ae1-5b36c812e296\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " Oct 31 17:42:28.473485 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.473288 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-tmp-dir\") pod \"baeaac10-b796-4dcd-9d24-752c346dff6a\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " Oct 31 17:42:28.473485 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.473323 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-dshm\") pod \"b962c13d-7616-4971-9ae1-5b36c812e296\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " Oct 31 17:42:28.473485 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.473409 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-model-cache\") pod \"baeaac10-b796-4dcd-9d24-752c346dff6a\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " Oct 31 17:42:28.473485 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.473455 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-model-cache\") pod \"b962c13d-7616-4971-9ae1-5b36c812e296\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " Oct 31 17:42:28.473485 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.473471 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baeaac10-b796-4dcd-9d24-752c346dff6a-kube-api-access-m77qx\" (UniqueName: \"kubernetes.io/projected/baeaac10-b796-4dcd-9d24-752c346dff6a-kube-api-access-m77qx\") pod \"baeaac10-b796-4dcd-9d24-752c346dff6a\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " Oct 31 17:42:28.473725 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.473492 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baeaac10-b796-4dcd-9d24-752c346dff6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/baeaac10-b796-4dcd-9d24-752c346dff6a-tls-certs\") pod \"baeaac10-b796-4dcd-9d24-752c346dff6a\" (UID: \"baeaac10-b796-4dcd-9d24-752c346dff6a\") " Oct 31 17:42:28.473725 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.473575 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b962c13d-7616-4971-9ae1-5b36c812e296-tls-certs\" (UniqueName: \"kubernetes.io/secret/b962c13d-7616-4971-9ae1-5b36c812e296-tls-certs\") pod \"b962c13d-7616-4971-9ae1-5b36c812e296\" (UID: \"b962c13d-7616-4971-9ae1-5b36c812e296\") " Oct 31 17:42:28.474453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.474420 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-model-cache" pod "b962c13d-7616-4971-9ae1-5b36c812e296" (UID: "b962c13d-7616-4971-9ae1-5b36c812e296"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:28.474595 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.474444 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-tmp-dir" pod "baeaac10-b796-4dcd-9d24-752c346dff6a" (UID: "baeaac10-b796-4dcd-9d24-752c346dff6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:28.474839 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.474813 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-tmp-dir" pod "b962c13d-7616-4971-9ae1-5b36c812e296" (UID: "b962c13d-7616-4971-9ae1-5b36c812e296"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:28.474926 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.474846 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-model-cache" pod "baeaac10-b796-4dcd-9d24-752c346dff6a" (UID: "baeaac10-b796-4dcd-9d24-752c346dff6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:28.474997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.474972 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-home" pod "baeaac10-b796-4dcd-9d24-752c346dff6a" (UID: "baeaac10-b796-4dcd-9d24-752c346dff6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:28.475094 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.475070 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-home" pod "b962c13d-7616-4971-9ae1-5b36c812e296" (UID: "b962c13d-7616-4971-9ae1-5b36c812e296"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:28.476849 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.476830 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baeaac10-b796-4dcd-9d24-752c346dff6a-tls-certs" pod "baeaac10-b796-4dcd-9d24-752c346dff6a" (UID: "baeaac10-b796-4dcd-9d24-752c346dff6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:28.477156 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.476851 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baeaac10-b796-4dcd-9d24-752c346dff6a-kube-api-access-m77qx" pod "baeaac10-b796-4dcd-9d24-752c346dff6a" (UID: "baeaac10-b796-4dcd-9d24-752c346dff6a"). InnerVolumeSpecName "kube-api-access-m77qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:28.477323 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.477026 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b962c13d-7616-4971-9ae1-5b36c812e296-tls-certs" pod "b962c13d-7616-4971-9ae1-5b36c812e296" (UID: "b962c13d-7616-4971-9ae1-5b36c812e296"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:28.477440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.477099 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-dshm" pod "baeaac10-b796-4dcd-9d24-752c346dff6a" (UID: "baeaac10-b796-4dcd-9d24-752c346dff6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:28.477537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.477110 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b962c13d-7616-4971-9ae1-5b36c812e296-kube-api-access-jhsc7" pod "b962c13d-7616-4971-9ae1-5b36c812e296" (UID: "b962c13d-7616-4971-9ae1-5b36c812e296"). InnerVolumeSpecName "kube-api-access-jhsc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:28.478320 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.478299 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-dshm" pod "b962c13d-7616-4971-9ae1-5b36c812e296" (UID: "b962c13d-7616-4971-9ae1-5b36c812e296"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:28.575613 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.575506 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:28.575613 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.575549 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:28.575613 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.575563 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:28.575613 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.575580 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:28.575613 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.575593 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:28.575613 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.575606 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:28.576132 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.575658 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m77qx\" (UniqueName: \"kubernetes.io/projected/baeaac10-b796-4dcd-9d24-752c346dff6a-kube-api-access-m77qx\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:28.576132 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.575675 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baeaac10-b796-4dcd-9d24-752c346dff6a-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:28.576132 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.575689 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b962c13d-7616-4971-9ae1-5b36c812e296-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:28.576132 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.575703 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhsc7\" (UniqueName: \"kubernetes.io/projected/b962c13d-7616-4971-9ae1-5b36c812e296-kube-api-access-jhsc7\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:28.576132 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.575717 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeaac10-b796-4dcd-9d24-752c346dff6a-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:28.576132 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.575731 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b962c13d-7616-4971-9ae1-5b36c812e296-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:28.669364 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.669323 7618 generic.go:356] "Generic (PLEG): container finished" podID="b962c13d-7616-4971-9ae1-5b36c812e296" containerID="8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4" exitCode=0 Oct 31 17:42:28.669576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.669393 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" Oct 31 17:42:28.669576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.669427 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" event={"ID":"b962c13d-7616-4971-9ae1-5b36c812e296","Type":"ContainerDied","Data":"8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4"} Oct 31 17:42:28.669576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.669476 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq" event={"ID":"b962c13d-7616-4971-9ae1-5b36c812e296","Type":"ContainerDied","Data":"637737e0a7edd40a4b2d4c3433b35e8fb40e394073b38b09a14a5ce23a9b86e0"} Oct 31 17:42:28.669576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.669496 7618 scope.go:122] "RemoveContainer" containerID="8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4" Oct 31 17:42:28.671747 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.671721 7618 generic.go:356] "Generic (PLEG): container finished" podID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerID="f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940" exitCode=0 Oct 31 17:42:28.671878 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.671781 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" event={"ID":"baeaac10-b796-4dcd-9d24-752c346dff6a","Type":"ContainerDied","Data":"f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940"} Oct 31 17:42:28.671878 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.671808 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" event={"ID":"baeaac10-b796-4dcd-9d24-752c346dff6a","Type":"ContainerDied","Data":"46b063fe10b6b16a47609250ddc611e138e94c3e1db4a5af2ee4212efd7c3000"} Oct 31 17:42:28.671878 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.671859 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f" Oct 31 17:42:28.683200 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.683176 7618 scope.go:122] "RemoveContainer" containerID="8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4" Oct 31 17:42:28.683671 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:28.683605 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4\": container with ID starting with 8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4 not found: ID does not exist" containerID="8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4" Oct 31 17:42:28.683671 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.683648 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4"} err="failed to get container status \"8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4\": rpc error: code = NotFound desc = could not find container \"8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4\": container with ID starting with 8d61be2f8189d23d8f4c8abf077d53ad8796b9997ef753d8f1cba5c5a9760ad4 not found: ID does not exist" Oct 31 17:42:28.683671 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.683665 7618 scope.go:122] "RemoveContainer" containerID="f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940" Oct 31 17:42:28.697937 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.697911 7618 scope.go:122] "RemoveContainer" containerID="f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940" Oct 31 17:42:28.698374 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:28.698348 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940\": container with ID starting with f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940 not found: ID does not exist" containerID="f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940" Oct 31 17:42:28.698435 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.698386 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940"} err="failed to get container status \"f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940\": rpc error: code = NotFound desc = could not find container \"f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940\": container with ID starting with f7043ae47c01bb15291c8f1ea0e7ab158ab931d2b94951806b1d3faa588c5940 not found: ID does not exist" Oct 31 17:42:28.732424 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.732384 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq"] Oct 31 17:42:28.741675 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.741636 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0e59ca/distinct-w1-0e59ca-kserve-8544d68ddf-9zntq"] Oct 31 17:42:28.763591 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.763547 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f"] Oct 31 17:42:28.767592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:28.767564 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0e59ca/distinct-2-w1-0e59ca-kserve-dc79d4586-7dw2f"] Oct 31 17:42:29.073112 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073079 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp"] Oct 31 17:42:29.073769 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073752 7618 cpu_manager.go:414] "containerMap: removing container" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" Oct 31 17:42:29.073826 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073770 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" Oct 31 17:42:29.073826 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073797 7618 cpu_manager.go:414] "containerMap: removing container" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" Oct 31 17:42:29.073826 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073802 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" Oct 31 17:42:29.073826 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073816 7618 cpu_manager.go:414] "containerMap: removing container" podUID="54de558b-f489-46a4-8978-de66c15583d7" containerName="revoke-keys" Oct 31 17:42:29.073826 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073821 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54de558b-f489-46a4-8978-de66c15583d7" containerName="revoke-keys" Oct 31 17:42:29.073966 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073833 7618 cpu_manager.go:414] "containerMap: removing container" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" Oct 31 17:42:29.073966 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073838 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" Oct 31 17:42:29.073966 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073849 7618 cpu_manager.go:414] "containerMap: removing container" podUID="e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7" containerName="revoke-keys" Oct 31 17:42:29.073966 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073854 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7" containerName="revoke-keys" Oct 31 17:42:29.073966 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073949 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" containerName="main" Oct 31 17:42:29.073966 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073960 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="54de558b-f489-46a4-8978-de66c15583d7" containerName="revoke-keys" Oct 31 17:42:29.073966 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073967 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" containerName="main" Oct 31 17:42:29.074157 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073974 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7" containerName="revoke-keys" Oct 31 17:42:29.074157 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.073984 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b13cb1-a3f6-4421-b295-547e34fd90be" containerName="main" Oct 31 17:42:29.079228 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.079205 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:29.094625 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.094586 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:42:29.094625 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.094597 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:42:29.095790 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.095760 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp"] Oct 31 17:42:29.184609 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.184557 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10617ae3-rcvsp\" (UID: \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:29.184786 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.184615 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10617ae3-rcvsp\" (UID: \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:29.184786 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.184727 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n5qp\" (UniqueName: \"kubernetes.io/projected/ab4fbb38-4cfa-4aa6-9921-4516f153231f-kube-api-access-6n5qp\") pod \"maas-api-revoke-sub-10617ae3-rcvsp\" (UID: \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:29.286054 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.285948 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10617ae3-rcvsp\" (UID: \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:29.286054 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.285994 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10617ae3-rcvsp\" (UID: \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:29.286054 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.286035 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n5qp\" (UniqueName: \"kubernetes.io/projected/ab4fbb38-4cfa-4aa6-9921-4516f153231f-kube-api-access-6n5qp\") pod \"maas-api-revoke-sub-10617ae3-rcvsp\" (UID: \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:29.286926 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.286808 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10617ae3-rcvsp\" (UID: \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:29.288682 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.288652 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10617ae3-rcvsp\" (UID: \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:29.304384 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.304352 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n5qp\" (UniqueName: \"kubernetes.io/projected/ab4fbb38-4cfa-4aa6-9921-4516f153231f-kube-api-access-6n5qp\") pod \"maas-api-revoke-sub-10617ae3-rcvsp\" (UID: \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:29.395223 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.395186 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:29.912403 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.912364 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8"] Oct 31 17:42:29.951293 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.950619 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8"] Oct 31 17:42:29.976415 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:29.961544 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:30.070156 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:42:30.070117 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4fbb38_4cfa_4aa6_9921_4516f153231f.slice/crio-9179b5b036a178432648ff1c87ca2b38a7f377e48cb14b5e1fc1ad1b3252b6f5 WatchSource:0}: Error finding container 9179b5b036a178432648ff1c87ca2b38a7f377e48cb14b5e1fc1ad1b3252b6f5: Status 404 returned error can't find the container with id 9179b5b036a178432648ff1c87ca2b38a7f377e48cb14b5e1fc1ad1b3252b6f5 Oct 31 17:42:30.091300 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.091232 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp"] Oct 31 17:42:30.137170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.129102 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64j4t\" (UniqueName: \"kubernetes.io/projected/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-kube-api-access-64j4t\") pod \"maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8\" (UID: \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:30.137170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.129209 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8\" (UID: \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:30.137170 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.129369 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8\" (UID: \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:30.233169 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.232235 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8\" (UID: \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:30.233169 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.232426 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8\" (UID: \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:30.233169 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.232622 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64j4t\" (UniqueName: \"kubernetes.io/projected/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-kube-api-access-64j4t\") pod \"maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8\" (UID: \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:30.233169 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.233069 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8\" (UID: \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:30.250308 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.243989 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8\" (UID: \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:30.278700 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.278432 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64j4t\" (UniqueName: \"kubernetes.io/projected/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-kube-api-access-64j4t\") pod \"maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8\" (UID: \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:30.286081 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.286050 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:30.394716 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.393171 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b962c13d-7616-4971-9ae1-5b36c812e296" path="/var/lib/kubelet/pods/b962c13d-7616-4971-9ae1-5b36c812e296/volumes" Oct 31 17:42:30.394716 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.393730 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baeaac10-b796-4dcd-9d24-752c346dff6a" path="/var/lib/kubelet/pods/baeaac10-b796-4dcd-9d24-752c346dff6a/volumes" Oct 31 17:42:30.462983 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.454901 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt"] Oct 31 17:42:30.462983 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.461195 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cf0a86-rgcvt"] Oct 31 17:42:30.672947 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.672844 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8"] Oct 31 17:42:30.891711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.891637 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2"] Oct 31 17:42:30.907402 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.906057 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c446c-dscq2"] Oct 31 17:42:30.988655 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.988529 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" event={"ID":"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1","Type":"ContainerStarted","Data":"8733e738351cfb45f779519addab707c27b088515f95ad0c5855fb6d1a7d58a2"} Oct 31 17:42:30.991576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.991518 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" event={"ID":"ab4fbb38-4cfa-4aa6-9921-4516f153231f","Type":"ContainerStarted","Data":"fb93d62c1649d8a86201af24b5a407d4e338e07e043ef983c9b46d0bcdb9cf40"} Oct 31 17:42:30.991905 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:30.991881 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" event={"ID":"ab4fbb38-4cfa-4aa6-9921-4516f153231f","Type":"ContainerStarted","Data":"9179b5b036a178432648ff1c87ca2b38a7f377e48cb14b5e1fc1ad1b3252b6f5"} Oct 31 17:42:31.062913 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:31.062837 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" podStartSLOduration=2.062813538 podStartE2EDuration="2.062813538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:42:31.026783708 +0000 UTC m=+872.774758176" watchObservedRunningTime="2026-10-31 17:42:31.062813538 +0000 UTC m=+872.810788013" Oct 31 17:42:31.977274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:31.976337 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:42:32.008447 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:32.008345 7618 generic.go:356] "Generic (PLEG): container finished" podID="c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1" containerID="b47e26d9bcc2ba03d671959f0893e17f2c2a72097f5fc73bf206ca165b218e29" exitCode=0 Oct 31 17:42:32.008447 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:32.008428 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" event={"ID":"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1","Type":"ContainerDied","Data":"b47e26d9bcc2ba03d671959f0893e17f2c2a72097f5fc73bf206ca165b218e29"} Oct 31 17:42:32.013641 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:32.010412 7618 generic.go:356] "Generic (PLEG): container finished" podID="ab4fbb38-4cfa-4aa6-9921-4516f153231f" containerID="fb93d62c1649d8a86201af24b5a407d4e338e07e043ef983c9b46d0bcdb9cf40" exitCode=0 Oct 31 17:42:32.013641 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:32.010496 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" event={"ID":"ab4fbb38-4cfa-4aa6-9921-4516f153231f","Type":"ContainerDied","Data":"fb93d62c1649d8a86201af24b5a407d4e338e07e043ef983c9b46d0bcdb9cf40"} Oct 31 17:42:32.581944 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:32.581893 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67eefead-6f47-43da-88ca-3b5e2780e39e" path="/var/lib/kubelet/pods/67eefead-6f47-43da-88ca-3b5e2780e39e/volumes" Oct 31 17:42:32.582749 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:32.582667 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a26f47-a048-453c-8950-36c92b30520c" path="/var/lib/kubelet/pods/88a26f47-a048-453c-8950-36c92b30520c/volumes" Oct 31 17:42:34.265332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.250339 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:34.307277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.296016 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" event={"ID":"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1","Type":"ContainerDied","Data":"8733e738351cfb45f779519addab707c27b088515f95ad0c5855fb6d1a7d58a2"} Oct 31 17:42:34.307277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.296086 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8733e738351cfb45f779519addab707c27b088515f95ad0c5855fb6d1a7d58a2" Oct 31 17:42:34.312631 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.312600 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:34.347197 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.347160 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-openshift-ca\") pod \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\" (UID: \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\") " Oct 31 17:42:34.347530 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.347492 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-xks-ca\") pod \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\" (UID: \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\") " Oct 31 17:42:34.347848 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.347829 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-kube-api-access-64j4t\" (UniqueName: \"kubernetes.io/projected/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-kube-api-access-64j4t\") pod \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\" (UID: \"c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1\") " Oct 31 17:42:34.350598 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.350288 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-openshift-ca" pod "c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1" (UID: "c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:34.351295 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.351270 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:34.408294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.400901 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-kube-api-access-64j4t" pod "c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1" (UID: "c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1"). InnerVolumeSpecName "kube-api-access-64j4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:34.408294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.402534 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-xks-ca" pod "c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1" (UID: "c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:34.408558 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.408334 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:34.481277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.480522 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4fbb38-4cfa-4aa6-9921-4516f153231f-kube-api-access-6n5qp\" (UniqueName: \"kubernetes.io/projected/ab4fbb38-4cfa-4aa6-9921-4516f153231f-kube-api-access-6n5qp\") pod \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\" (UID: \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\") " Oct 31 17:42:34.481277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.480799 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-openshift-ca\") pod \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\" (UID: \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\") " Oct 31 17:42:34.481277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.481004 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-xks-ca\") pod \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\" (UID: \"ab4fbb38-4cfa-4aa6-9921-4516f153231f\") " Oct 31 17:42:34.482110 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.482088 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:34.482276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.482240 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:34.482396 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.482381 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64j4t\" (UniqueName: \"kubernetes.io/projected/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1-kube-api-access-64j4t\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:34.486549 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.486515 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-openshift-ca" pod "ab4fbb38-4cfa-4aa6-9921-4516f153231f" (UID: "ab4fbb38-4cfa-4aa6-9921-4516f153231f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:34.599861 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.599765 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:34.681287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.673538 7618 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 31 17:42:34.734833 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.733578 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-xks-ca" pod "ab4fbb38-4cfa-4aa6-9921-4516f153231f" (UID: "ab4fbb38-4cfa-4aa6-9921-4516f153231f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:34.734833 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.734427 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4fbb38-4cfa-4aa6-9921-4516f153231f-kube-api-access-6n5qp" pod "ab4fbb38-4cfa-4aa6-9921-4516f153231f" (UID: "ab4fbb38-4cfa-4aa6-9921-4516f153231f"). InnerVolumeSpecName "kube-api-access-6n5qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:34.735521 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.735491 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab4fbb38-4cfa-4aa6-9921-4516f153231f-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:34.735723 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:34.735707 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n5qp\" (UniqueName: \"kubernetes.io/projected/ab4fbb38-4cfa-4aa6-9921-4516f153231f-kube-api-access-6n5qp\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:35.095888 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.078395 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr"] Oct 31 17:42:35.128639 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.128586 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:35.207692 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.207332 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr"] Oct 31 17:42:35.218814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.218432 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:35.264910 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.257557 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q"] Oct 31 17:42:35.266284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.266227 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 17:42:35.267203 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.267154 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:42:35.376793 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.376748 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 17:42:35.406515 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.402619 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:35.432277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.420367 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 17:42:35.433236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.432953 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:42:35.460353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.435571 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8" Oct 31 17:42:35.460353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.443870 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" Oct 31 17:42:35.462390 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.460831 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp" event={"ID":"ab4fbb38-4cfa-4aa6-9921-4516f153231f","Type":"ContainerDied","Data":"9179b5b036a178432648ff1c87ca2b38a7f377e48cb14b5e1fc1ad1b3252b6f5"} Oct 31 17:42:35.462390 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:35.460902 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9179b5b036a178432648ff1c87ca2b38a7f377e48cb14b5e1fc1ad1b3252b6f5" Oct 31 17:42:36.226604 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:36.162895 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:36.226604 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:36.166085 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:42:36.227053 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:36.227018 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 17:42:36.232274 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:36.228143 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:42:36.336577 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:36.289508 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:36.725208 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:36.725154 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:36.758565 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:36.750756 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:37.166388 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:37.166336 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 31 17:42:37.277331 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:37.251859 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:37.291825 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:37.291767 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 17:42:37.346311 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:37.346276 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:42:37.788279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:37.780409 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:37.794039 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:37.793981 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:37.802304 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:37.798212 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:38.167634 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.167587 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:38.216434 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.216383 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" podUID="8c01da1b-48fc-49e4-9341-54965a5c45a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 17:42:38.279283 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.278566 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:38.304393 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.304342 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:38.704435 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.704392 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk"] Oct 31 17:42:38.710163 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.710117 7618 cpu_manager.go:414] "containerMap: removing container" podUID="c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1" containerName="revoke-keys" Oct 31 17:42:38.713473 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.713445 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1" containerName="revoke-keys" Oct 31 17:42:38.713788 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.713765 7618 cpu_manager.go:414] "containerMap: removing container" podUID="ab4fbb38-4cfa-4aa6-9921-4516f153231f" containerName="revoke-keys" Oct 31 17:42:38.713923 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.713907 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4fbb38-4cfa-4aa6-9921-4516f153231f" containerName="revoke-keys" Oct 31 17:42:38.722677 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.722643 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4fbb38-4cfa-4aa6-9921-4516f153231f" containerName="revoke-keys" Oct 31 17:42:38.722925 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.722905 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1" containerName="revoke-keys" Oct 31 17:42:38.758897 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.758859 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:38.782230 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.782186 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 17:42:38.782993 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.782967 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:42:38.783246 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.783229 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:42:38.805436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.805393 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 31 17:42:38.811603 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.806368 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 17:42:38.847708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.847671 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:42:38.936165 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:38.933719 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk"] Oct 31 17:42:39.036130 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.036027 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e4b5981-k6cdk\" (UID: \"c586ef82-a0d9-4dcb-8504-339d26c88b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:39.036596 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.036571 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnxgd\" (UniqueName: \"kubernetes.io/projected/c586ef82-a0d9-4dcb-8504-339d26c88b04-kube-api-access-xnxgd\") pod \"maas-api-revoke-sub-6e4b5981-k6cdk\" (UID: \"c586ef82-a0d9-4dcb-8504-339d26c88b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:39.036756 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.036739 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e4b5981-k6cdk\" (UID: \"c586ef82-a0d9-4dcb-8504-339d26c88b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:39.179354 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.164474 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnxgd\" (UniqueName: \"kubernetes.io/projected/c586ef82-a0d9-4dcb-8504-339d26c88b04-kube-api-access-xnxgd\") pod \"maas-api-revoke-sub-6e4b5981-k6cdk\" (UID: \"c586ef82-a0d9-4dcb-8504-339d26c88b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:39.179354 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.164539 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e4b5981-k6cdk\" (UID: \"c586ef82-a0d9-4dcb-8504-339d26c88b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:39.179354 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.164678 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e4b5981-k6cdk\" (UID: \"c586ef82-a0d9-4dcb-8504-339d26c88b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:39.192516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.192480 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e4b5981-k6cdk\" (UID: \"c586ef82-a0d9-4dcb-8504-339d26c88b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:39.203367 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.200053 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e4b5981-k6cdk\" (UID: \"c586ef82-a0d9-4dcb-8504-339d26c88b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:39.269781 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.204432 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 31 17:42:39.414065 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:39.414016 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:42:39.433396 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.433346 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 17:42:39.471534 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.453487 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:42:39.471534 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.455732 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnxgd\" (UniqueName: \"kubernetes.io/projected/c586ef82-a0d9-4dcb-8504-339d26c88b04-kube-api-access-xnxgd\") pod \"maas-api-revoke-sub-6e4b5981-k6cdk\" (UID: \"c586ef82-a0d9-4dcb-8504-339d26c88b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:39.516652 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.516614 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc"] Oct 31 17:42:39.518688 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.518647 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" containerID="cri-o://c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05" gracePeriod=30 Oct 31 17:42:39.547158 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.547114 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv"] Oct 31 17:42:39.547485 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.547455 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" containerID="cri-o://e3bd2e57f25fd5c2681fb5e95ac088666da48ea6424acfdd1f556476d7b4bbbd" gracePeriod=30 Oct 31 17:42:39.576372 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.576323 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg"] Oct 31 17:42:39.578520 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.578475 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" containerName="istio-proxy" containerID="cri-o://73c514d6dac9e1b2ff0750f882859e26e9a6a3a7d6ec528d7f56b5827e8ab079" gracePeriod=30 Oct 31 17:42:39.700513 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.700424 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" containerID="cri-o://75da3d280119c73a000df361198ad8877d13506b2b82195e76f1414210349bf4" gracePeriod=45 Oct 31 17:42:39.701523 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.701439 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:39.754405 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.754370 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht"] Oct 31 17:42:39.805299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.799021 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:39.811570 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.811534 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" containerID="cri-o://25b33c0ea918c5c296923cb846b4001154c30b29bff7679599c59150e4ad6f70" gracePeriod=30 Oct 31 17:42:39.824905 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.806236 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:39.879092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.867880 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v"] Oct 31 17:42:39.879092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.869008 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 17:42:39.879092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:39.868189 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerName="istio-proxy" containerID="cri-o://cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1" gracePeriod=30 Oct 31 17:42:40.004806 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:40.004712 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5"] Oct 31 17:42:40.005278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:40.005229 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" containerID="cri-o://8387da5b989ca9d565255372597055dbe84b41509cf4ad5935c7898570e69b8e" gracePeriod=30 Oct 31 17:42:40.210401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:40.210348 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:40.243367 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:40.243313 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused" Oct 31 17:42:40.434852 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:40.434788 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:40.813611 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:40.812965 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 17:42:40.858588 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:40.849356 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:40.877963 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:40.877913 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:41.066827 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.066422 7618 patch_prober.go:29] interesting pod/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:42:41.066827 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.066487 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:42:41.215301 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.195397 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:42:41.215301 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.195466 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" podUID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:41.215301 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.198357 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:42:41.215301 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.198420 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" podUID="446910ca-78af-4c6f-8196-56c209b33636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:41.222772 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.215979 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 31 17:42:41.240137 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.240087 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused" Oct 31 17:42:41.251283 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.240482 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:42:41.311678 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.303507 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" containerID="cri-o://d5f5747c0474a8e49f264dd8bcddee1ad1a6b7c813c000536681d92469559eda" gracePeriod=45 Oct 31 17:42:41.358350 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.318604 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:42:41.358350 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.348704 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:42:41.358350 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.348764 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:42:41.450474 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.448386 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:41.739936 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.735552 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:42:41.739936 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.735627 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:41.850952 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.824373 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:41.903755 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.896656 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:41.916323 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.916289 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:42:41.940276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.936621 7618 patch_prober.go:29] interesting pod/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:42:41.940276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:41.936694 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:42:42.018333 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:42:42.002575 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc586ef82_a0d9_4dcb_8504_339d26c88b04.slice/crio-7beba9d458a3c923cb2c330339106cf5bf75741c83e48c32e01de9174b69767f WatchSource:0}: Error finding container 7beba9d458a3c923cb2c330339106cf5bf75741c83e48c32e01de9174b69767f: Status 404 returned error can't find the container with id 7beba9d458a3c923cb2c330339106cf5bf75741c83e48c32e01de9174b69767f Oct 31 17:42:42.054433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.054382 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": read tcp 10.134.0.2:47432->10.134.0.46:8000: read: connection reset by peer" Oct 31 17:42:42.161383 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.161345 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk"] Oct 31 17:42:42.260412 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.260364 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:42.315976 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.315699 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused" Oct 31 17:42:42.353028 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.352910 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": dial tcp 10.134.0.46:8000: connect: connection refused" Oct 31 17:42:42.425771 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.425724 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:42.504488 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.504444 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:42.804483 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.804422 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:42:42.804787 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.804745 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:42.914606 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.914574 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:42:42.917042 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.917009 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" event={"ID":"c586ef82-a0d9-4dcb-8504-339d26c88b04","Type":"ContainerStarted","Data":"7beba9d458a3c923cb2c330339106cf5bf75741c83e48c32e01de9174b69767f"} Oct 31 17:42:42.921328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.921294 7618 generic.go:356] "Generic (PLEG): container finished" podID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerID="75da3d280119c73a000df361198ad8877d13506b2b82195e76f1414210349bf4" exitCode=0 Oct 31 17:42:42.921628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.921603 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" event={"ID":"5803fb07-04c1-4d98-8278-b0f21077d21b","Type":"ContainerDied","Data":"75da3d280119c73a000df361198ad8877d13506b2b82195e76f1414210349bf4"} Oct 31 17:42:42.926215 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.926184 7618 generic.go:356] "Generic (PLEG): container finished" podID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerID="d5f5747c0474a8e49f264dd8bcddee1ad1a6b7c813c000536681d92469559eda" exitCode=0 Oct 31 17:42:42.926711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:42.926423 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" event={"ID":"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd","Type":"ContainerDied","Data":"d5f5747c0474a8e49f264dd8bcddee1ad1a6b7c813c000536681d92469559eda"} Oct 31 17:42:43.217328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:43.212872 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" containerID="cri-o://0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d" gracePeriod=45 Oct 31 17:42:43.242209 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:43.242111 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused" Oct 31 17:42:43.343902 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:43.343860 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": dial tcp 10.134.0.46:8000: connect: connection refused" Oct 31 17:42:43.426052 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:43.426003 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": read tcp 10.134.0.2:50008->10.134.0.75:8000: read: connection reset by peer" Oct 31 17:42:43.535406 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:43.511363 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:43.866397 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:43.866366 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:42:44.223420 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.223378 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" event={"ID":"c586ef82-a0d9-4dcb-8504-339d26c88b04","Type":"ContainerStarted","Data":"fcd00a5f5e84d3729dd1b0e002cf2cbd47f794513567ff1e92b1b244578b5add"} Oct 31 17:42:44.227189 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.227159 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" event={"ID":"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd","Type":"ContainerDied","Data":"273266ef794934e658c79cff71e4502abb5fdc82ae822d576a755132a97362ca"} Oct 31 17:42:44.227398 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.227381 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="273266ef794934e658c79cff71e4502abb5fdc82ae822d576a755132a97362ca" Oct 31 17:42:44.230534 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.230508 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:42:44.273281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.271835 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" podStartSLOduration=7.271814381 podStartE2EDuration="7.271814381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:42:44.267590452 +0000 UTC m=+886.015564921" watchObservedRunningTime="2026-10-31 17:42:44.271814381 +0000 UTC m=+886.019788849" Oct 31 17:42:44.343866 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.342494 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.382861 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tls-certs\") pod \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.382915 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-home\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-home\") pod \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.382946 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-kube-api-access-q78j7\" (UniqueName: \"kubernetes.io/projected/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-kube-api-access-q78j7\") pod \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.383229 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tmp-dir\") pod \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.383292 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-model-cache\") pod \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.383342 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-dshm\") pod \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\" (UID: \"b4f22d3d-03f9-495b-9b3e-bbc3aea680fd\") " Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.385922 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-home" pod "b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" (UID: "b4f22d3d-03f9-495b-9b3e-bbc3aea680fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.386677 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tmp-dir" pod "b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" (UID: "b4f22d3d-03f9-495b-9b3e-bbc3aea680fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.388984 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-dshm" pod "b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" (UID: "b4f22d3d-03f9-495b-9b3e-bbc3aea680fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.389791 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-kube-api-access-q78j7" pod "b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" (UID: "b4f22d3d-03f9-495b-9b3e-bbc3aea680fd"). InnerVolumeSpecName "kube-api-access-q78j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.391243 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": dial tcp 10.134.0.75:8000: connect: connection refused" Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.394684 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-model-cache" pod "b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" (UID: "b4f22d3d-03f9-495b-9b3e-bbc3aea680fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:44.395298 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.394980 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tls-certs" pod "b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" (UID: "b4f22d3d-03f9-495b-9b3e-bbc3aea680fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:44.409275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.408981 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:42:44.485181 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.485142 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5803fb07-04c1-4d98-8278-b0f21077d21b-tls-certs\" (UniqueName: \"kubernetes.io/secret/5803fb07-04c1-4d98-8278-b0f21077d21b-tls-certs\") pod \"5803fb07-04c1-4d98-8278-b0f21077d21b\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " Oct 31 17:42:44.485357 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.485312 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-home\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-home\") pod \"5803fb07-04c1-4d98-8278-b0f21077d21b\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " Oct 31 17:42:44.485445 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.485427 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-dshm\") pod \"5803fb07-04c1-4d98-8278-b0f21077d21b\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " Oct 31 17:42:44.485527 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.485512 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-tmp-dir\") pod \"5803fb07-04c1-4d98-8278-b0f21077d21b\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " Oct 31 17:42:44.485598 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.485582 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5803fb07-04c1-4d98-8278-b0f21077d21b-kube-api-access-6r5bj\" (UniqueName: \"kubernetes.io/projected/5803fb07-04c1-4d98-8278-b0f21077d21b-kube-api-access-6r5bj\") pod \"5803fb07-04c1-4d98-8278-b0f21077d21b\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " Oct 31 17:42:44.485666 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.485615 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-model-cache\") pod \"5803fb07-04c1-4d98-8278-b0f21077d21b\" (UID: \"5803fb07-04c1-4d98-8278-b0f21077d21b\") " Oct 31 17:42:44.486814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.486135 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-home" pod "5803fb07-04c1-4d98-8278-b0f21077d21b" (UID: "5803fb07-04c1-4d98-8278-b0f21077d21b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:44.486814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.486477 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.486814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.486499 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.486814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.486514 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q78j7\" (UniqueName: \"kubernetes.io/projected/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-kube-api-access-q78j7\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.486814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.486526 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.486814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.486539 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.486814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.486552 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.486814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.486564 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.486814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.486768 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-model-cache" pod "5803fb07-04c1-4d98-8278-b0f21077d21b" (UID: "5803fb07-04c1-4d98-8278-b0f21077d21b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:44.487554 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.487243 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-tmp-dir" pod "5803fb07-04c1-4d98-8278-b0f21077d21b" (UID: "5803fb07-04c1-4d98-8278-b0f21077d21b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:44.488426 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.488399 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-dshm" pod "5803fb07-04c1-4d98-8278-b0f21077d21b" (UID: "5803fb07-04c1-4d98-8278-b0f21077d21b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:44.488895 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.488866 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5803fb07-04c1-4d98-8278-b0f21077d21b-kube-api-access-6r5bj" pod "5803fb07-04c1-4d98-8278-b0f21077d21b" (UID: "5803fb07-04c1-4d98-8278-b0f21077d21b"). InnerVolumeSpecName "kube-api-access-6r5bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:44.490101 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.490076 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5803fb07-04c1-4d98-8278-b0f21077d21b-tls-certs" pod "5803fb07-04c1-4d98-8278-b0f21077d21b" (UID: "5803fb07-04c1-4d98-8278-b0f21077d21b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:44.572194 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.568047 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz"] Oct 31 17:42:44.572194 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.569314 7618 cpu_manager.go:414] "containerMap: removing container" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" Oct 31 17:42:44.572194 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.569337 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" Oct 31 17:42:44.572194 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.569365 7618 cpu_manager.go:414] "containerMap: removing container" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" Oct 31 17:42:44.572194 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.569373 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" Oct 31 17:42:44.572194 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.569573 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" Oct 31 17:42:44.572194 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.569589 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" containerName="main" Oct 31 17:42:44.588276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.580235 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:44.588276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.588150 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.588276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.588175 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.588276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.588193 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r5bj\" (UniqueName: \"kubernetes.io/projected/5803fb07-04c1-4d98-8278-b0f21077d21b-kube-api-access-6r5bj\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.588276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.588210 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5803fb07-04c1-4d98-8278-b0f21077d21b-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.588276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.588225 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5803fb07-04c1-4d98-8278-b0f21077d21b-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.617216 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.615648 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz"] Oct 31 17:42:44.683517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.683478 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" Oct 31 17:42:44.689424 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.688968 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d81ce110-ghfjz\" (UID: \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:44.689424 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.689095 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d81ce110-ghfjz\" (UID: \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:44.689424 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.689129 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f5sw\" (UniqueName: \"kubernetes.io/projected/9d3288d2-a16e-4770-9e3e-b06584d07dd8-kube-api-access-4f5sw\") pod \"maas-api-revoke-sub-d81ce110-ghfjz\" (UID: \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:44.689710 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.689687 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:42:44.790999 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.790145 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-home\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-home\") pod \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " Oct 31 17:42:44.790999 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.790356 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-dshm\") pod \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " Oct 31 17:42:44.790999 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.790460 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tmp-dir\") pod \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " Oct 31 17:42:44.790999 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.790504 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tls-certs\") pod \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " Oct 31 17:42:44.790999 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.790548 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c01da1b-48fc-49e4-9341-54965a5c45a0-kube-api-access-drjf7\" (UniqueName: \"kubernetes.io/projected/8c01da1b-48fc-49e4-9341-54965a5c45a0-kube-api-access-drjf7\") pod \"8c01da1b-48fc-49e4-9341-54965a5c45a0\" (UID: \"8c01da1b-48fc-49e4-9341-54965a5c45a0\") " Oct 31 17:42:44.790999 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.790585 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-model-cache\") pod \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " Oct 31 17:42:44.790999 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.790626 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c01da1b-48fc-49e4-9341-54965a5c45a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c01da1b-48fc-49e4-9341-54965a5c45a0-plugins-config-volume\") pod \"8c01da1b-48fc-49e4-9341-54965a5c45a0\" (UID: \"8c01da1b-48fc-49e4-9341-54965a5c45a0\") " Oct 31 17:42:44.790999 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.790671 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c13943-5746-4c6e-a4d2-5c11e89156bd-kube-api-access-xmdkk\" (UniqueName: \"kubernetes.io/projected/f7c13943-5746-4c6e-a4d2-5c11e89156bd-kube-api-access-xmdkk\") pod \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\" (UID: \"f7c13943-5746-4c6e-a4d2-5c11e89156bd\") " Oct 31 17:42:44.791559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.791118 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d81ce110-ghfjz\" (UID: \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:44.791559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.791309 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tmp-dir" pod "f7c13943-5746-4c6e-a4d2-5c11e89156bd" (UID: "f7c13943-5746-4c6e-a4d2-5c11e89156bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:44.791559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.791357 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d81ce110-ghfjz\" (UID: \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:44.791559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.791368 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-model-cache" pod "f7c13943-5746-4c6e-a4d2-5c11e89156bd" (UID: "f7c13943-5746-4c6e-a4d2-5c11e89156bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:44.792168 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.792134 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d81ce110-ghfjz\" (UID: \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:44.792322 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.792194 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f5sw\" (UniqueName: \"kubernetes.io/projected/9d3288d2-a16e-4770-9e3e-b06584d07dd8-kube-api-access-4f5sw\") pod \"maas-api-revoke-sub-d81ce110-ghfjz\" (UID: \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:44.792945 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.792427 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c01da1b-48fc-49e4-9341-54965a5c45a0-plugins-config-volume" pod "8c01da1b-48fc-49e4-9341-54965a5c45a0" (UID: "8c01da1b-48fc-49e4-9341-54965a5c45a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:44.792945 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.792453 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-home" pod "f7c13943-5746-4c6e-a4d2-5c11e89156bd" (UID: "f7c13943-5746-4c6e-a4d2-5c11e89156bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:44.792945 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.792461 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.792945 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.792483 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.795634 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.795594 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d81ce110-ghfjz\" (UID: \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:44.795909 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.795859 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c01da1b-48fc-49e4-9341-54965a5c45a0-kube-api-access-drjf7" pod "8c01da1b-48fc-49e4-9341-54965a5c45a0" (UID: "8c01da1b-48fc-49e4-9341-54965a5c45a0"). InnerVolumeSpecName "kube-api-access-drjf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:44.796353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.796327 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tls-certs" pod "f7c13943-5746-4c6e-a4d2-5c11e89156bd" (UID: "f7c13943-5746-4c6e-a4d2-5c11e89156bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:44.796562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.796541 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c13943-5746-4c6e-a4d2-5c11e89156bd-kube-api-access-xmdkk" pod "f7c13943-5746-4c6e-a4d2-5c11e89156bd" (UID: "f7c13943-5746-4c6e-a4d2-5c11e89156bd"). InnerVolumeSpecName "kube-api-access-xmdkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:44.797047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.797024 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-dshm" pod "f7c13943-5746-4c6e-a4d2-5c11e89156bd" (UID: "f7c13943-5746-4c6e-a4d2-5c11e89156bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:44.811963 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.811894 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f5sw\" (UniqueName: \"kubernetes.io/projected/9d3288d2-a16e-4770-9e3e-b06584d07dd8-kube-api-access-4f5sw\") pod \"maas-api-revoke-sub-d81ce110-ghfjz\" (UID: \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:44.896642 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.896512 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.896642 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.896552 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c13943-5746-4c6e-a4d2-5c11e89156bd-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.896642 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.896569 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c13943-5746-4c6e-a4d2-5c11e89156bd-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.896642 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.896585 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drjf7\" (UniqueName: \"kubernetes.io/projected/8c01da1b-48fc-49e4-9341-54965a5c45a0-kube-api-access-drjf7\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.896642 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.896599 7618 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c01da1b-48fc-49e4-9341-54965a5c45a0-plugins-config-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.896642 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.896613 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmdkk\" (UniqueName: \"kubernetes.io/projected/f7c13943-5746-4c6e-a4d2-5c11e89156bd-kube-api-access-xmdkk\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:44.949129 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:44.948844 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:45.009559 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.009139 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:42:45.109916 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.109873 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-credential-socket\") pod \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " Oct 31 17:42:45.110096 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.110018 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-token\" (UniqueName: \"kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-token\") pod \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " Oct 31 17:42:45.110181 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.110164 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-data\") pod \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " Oct 31 17:42:45.110233 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.110206 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-kube-api-access-lh8n9\" (UniqueName: \"kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-kube-api-access-lh8n9\") pod \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " Oct 31 17:42:45.110337 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.110267 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-socket\") pod \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " Oct 31 17:42:45.110391 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.110376 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-podinfo\") pod \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " Oct 31 17:42:45.110481 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.110461 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-certs\") pod \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " Oct 31 17:42:45.110544 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.110522 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-envoy\") pod \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " Oct 31 17:42:45.110604 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.110585 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istiod-ca-cert\") pod \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\" (UID: \"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a\") " Oct 31 17:42:45.112129 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.112095 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istiod-ca-cert" pod "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" (UID: "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:45.124597 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.124538 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-data" pod "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" (UID: "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.138269 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.138196 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-certs" pod "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" (UID: "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.141021 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.140950 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-envoy" pod "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" (UID: "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.176277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.175833 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9"] Oct 31 17:42:45.197347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.176890 7618 cpu_manager.go:414] "containerMap: removing container" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" Oct 31 17:42:45.197347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.176915 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" Oct 31 17:42:45.197347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.176945 7618 cpu_manager.go:414] "containerMap: removing container" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" Oct 31 17:42:45.197347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.176953 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" Oct 31 17:42:45.197347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.176974 7618 cpu_manager.go:414] "containerMap: removing container" podUID="8c01da1b-48fc-49e4-9341-54965a5c45a0" containerName="payload-processing" Oct 31 17:42:45.197347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.176982 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c01da1b-48fc-49e4-9341-54965a5c45a0" containerName="payload-processing" Oct 31 17:42:45.197347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.177224 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c01da1b-48fc-49e4-9341-54965a5c45a0" containerName="payload-processing" Oct 31 17:42:45.197347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.177266 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerName="main" Oct 31 17:42:45.197347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.177283 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerName="istio-proxy" Oct 31 17:42:45.235292 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.198975 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-credential-socket" pod "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" (UID: "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.235292 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.199997 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-socket" pod "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" (UID: "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.254920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.245737 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.254920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.245769 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.254920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.245787 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.254920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.245815 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.254920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.245831 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.254920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.245844 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.264385 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.263149 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:42:45.308168 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.304629 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:45.328001 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.304525 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9"] Oct 31 17:42:45.354109 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.331305 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-token" pod "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" (UID: "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:45.354109 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.332117 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-podinfo" pod "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" (UID: "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:42:45.393394 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.356607 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpnnf\" (UniqueName: \"kubernetes.io/projected/dbf57bc3-4558-446e-96bb-05d057e94b3c-kube-api-access-dpnnf\") pod \"maas-api-revoke-sub-5fb6e677-g7bn9\" (UID: \"dbf57bc3-4558-446e-96bb-05d057e94b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:45.393394 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.356750 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb6e677-g7bn9\" (UID: \"dbf57bc3-4558-446e-96bb-05d057e94b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:45.393394 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.356886 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb6e677-g7bn9\" (UID: \"dbf57bc3-4558-446e-96bb-05d057e94b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:45.393394 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.356974 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.393394 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.356990 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.396063 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.395823 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-kube-api-access-lh8n9" pod "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" (UID: "a08f2d28-721a-4f5c-9e8e-db5aefafdd0a"). InnerVolumeSpecName "kube-api-access-lh8n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:45.417933 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.417631 7618 generic.go:356] "Generic (PLEG): container finished" podID="79109729-f8dc-44d2-9366-128216bc2eb6" containerID="e3bd2e57f25fd5c2681fb5e95ac088666da48ea6424acfdd1f556476d7b4bbbd" exitCode=0 Oct 31 17:42:45.417933 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.417796 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" event={"ID":"79109729-f8dc-44d2-9366-128216bc2eb6","Type":"ContainerDied","Data":"e3bd2e57f25fd5c2681fb5e95ac088666da48ea6424acfdd1f556476d7b4bbbd"} Oct 31 17:42:45.420814 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.420781 7618 generic.go:356] "Generic (PLEG): container finished" podID="8c01da1b-48fc-49e4-9341-54965a5c45a0" containerID="878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767" exitCode=0 Oct 31 17:42:45.421135 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.421111 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" event={"ID":"8c01da1b-48fc-49e4-9341-54965a5c45a0","Type":"ContainerDied","Data":"878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767"} Oct 31 17:42:45.421311 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.421293 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" event={"ID":"8c01da1b-48fc-49e4-9341-54965a5c45a0","Type":"ContainerDied","Data":"99532bf855a201498e039fb1cfbdbca29d3ab63dc53e4c13fa9a03ccb88093db"} Oct 31 17:42:45.421467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.421452 7618 scope.go:122] "RemoveContainer" containerID="878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767" Oct 31 17:42:45.421743 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.421725 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr" Oct 31 17:42:45.446666 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.445532 7618 generic.go:356] "Generic (PLEG): container finished" podID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" containerID="0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d" exitCode=0 Oct 31 17:42:45.446666 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.445701 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" event={"ID":"f7c13943-5746-4c6e-a4d2-5c11e89156bd","Type":"ContainerDied","Data":"0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d"} Oct 31 17:42:45.446666 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.445742 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" event={"ID":"f7c13943-5746-4c6e-a4d2-5c11e89156bd","Type":"ContainerDied","Data":"40fe569f579fa570bcb3108456aad32f98278e68e64c34cb436386b6e39b7fbf"} Oct 31 17:42:45.446666 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.445827 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t" Oct 31 17:42:45.456087 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.454108 7618 generic.go:356] "Generic (PLEG): container finished" podID="c586ef82-a0d9-4dcb-8504-339d26c88b04" containerID="fcd00a5f5e84d3729dd1b0e002cf2cbd47f794513567ff1e92b1b244578b5add" exitCode=0 Oct 31 17:42:45.456087 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.454208 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" event={"ID":"c586ef82-a0d9-4dcb-8504-339d26c88b04","Type":"ContainerDied","Data":"fcd00a5f5e84d3729dd1b0e002cf2cbd47f794513567ff1e92b1b244578b5add"} Oct 31 17:42:45.465131 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.463229 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb6e677-g7bn9\" (UID: \"dbf57bc3-4558-446e-96bb-05d057e94b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:45.465131 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.463395 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpnnf\" (UniqueName: \"kubernetes.io/projected/dbf57bc3-4558-446e-96bb-05d057e94b3c-kube-api-access-dpnnf\") pod \"maas-api-revoke-sub-5fb6e677-g7bn9\" (UID: \"dbf57bc3-4558-446e-96bb-05d057e94b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:45.465131 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.463523 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb6e677-g7bn9\" (UID: \"dbf57bc3-4558-446e-96bb-05d057e94b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:45.465131 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.463686 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh8n9\" (UniqueName: \"kubernetes.io/projected/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a-kube-api-access-lh8n9\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.465131 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.464530 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb6e677-g7bn9\" (UID: \"dbf57bc3-4558-446e-96bb-05d057e94b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:45.474351 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.472640 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" event={"ID":"5803fb07-04c1-4d98-8278-b0f21077d21b","Type":"ContainerDied","Data":"2a024f7488260bbc569de4e84e249945957f575fa91eeb73b3ac2a6f1ab70bd2"} Oct 31 17:42:45.474351 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.472787 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5" Oct 31 17:42:45.486312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.477413 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb6e677-g7bn9\" (UID: \"dbf57bc3-4558-446e-96bb-05d057e94b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:45.486312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.481355 7618 generic.go:356] "Generic (PLEG): container finished" podID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" containerID="c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05" exitCode=0 Oct 31 17:42:45.486312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.481481 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" event={"ID":"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a","Type":"ContainerDied","Data":"c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05"} Oct 31 17:42:45.486312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.481531 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" event={"ID":"a08f2d28-721a-4f5c-9e8e-db5aefafdd0a","Type":"ContainerDied","Data":"a40828529206ae06e4cefd52ad194f7dc90ac088dd83577efe51c008df8cf6e2"} Oct 31 17:42:45.486312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.481620 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc" Oct 31 17:42:45.486312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.482901 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:42:45.547964 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.547924 7618 generic.go:356] "Generic (PLEG): container finished" podID="8bd011f8-7a9d-4814-871b-8349744550a5" containerID="73c514d6dac9e1b2ff0750f882859e26e9a6a3a7d6ec528d7f56b5827e8ab079" exitCode=0 Oct 31 17:42:45.550532 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.550505 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl" Oct 31 17:42:45.555485 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.555436 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" event={"ID":"8bd011f8-7a9d-4814-871b-8349744550a5","Type":"ContainerDied","Data":"73c514d6dac9e1b2ff0750f882859e26e9a6a3a7d6ec528d7f56b5827e8ab079"} Oct 31 17:42:45.574913 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.574879 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-credential-socket\") pod \"8bd011f8-7a9d-4814-871b-8349744550a5\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " Oct 31 17:42:45.578078 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.578052 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-kube-api-access-b4vwj\" (UniqueName: \"kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-kube-api-access-b4vwj\") pod \"8bd011f8-7a9d-4814-871b-8349744550a5\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " Oct 31 17:42:45.581527 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.581499 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-envoy\") pod \"8bd011f8-7a9d-4814-871b-8349744550a5\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " Oct 31 17:42:45.585469 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.585445 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-certs\") pod \"8bd011f8-7a9d-4814-871b-8349744550a5\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " Oct 31 17:42:45.585715 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.585699 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bd011f8-7a9d-4814-871b-8349744550a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd011f8-7a9d-4814-871b-8349744550a5-istio-podinfo\") pod \"8bd011f8-7a9d-4814-871b-8349744550a5\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " Oct 31 17:42:45.585899 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.585882 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd011f8-7a9d-4814-871b-8349744550a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd011f8-7a9d-4814-871b-8349744550a5-istiod-ca-cert\") pod \"8bd011f8-7a9d-4814-871b-8349744550a5\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " Oct 31 17:42:45.586033 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.586021 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-istio-token\" (UniqueName: \"kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-istio-token\") pod \"8bd011f8-7a9d-4814-871b-8349744550a5\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " Oct 31 17:42:45.586240 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.586227 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-socket\") pod \"8bd011f8-7a9d-4814-871b-8349744550a5\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " Oct 31 17:42:45.587592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.587568 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-data\") pod \"8bd011f8-7a9d-4814-871b-8349744550a5\" (UID: \"8bd011f8-7a9d-4814-871b-8349744550a5\") " Oct 31 17:42:45.594696 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.577872 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-credential-socket" pod "8bd011f8-7a9d-4814-871b-8349744550a5" (UID: "8bd011f8-7a9d-4814-871b-8349744550a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.594900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.584198 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-kube-api-access-b4vwj" pod "8bd011f8-7a9d-4814-871b-8349744550a5" (UID: "8bd011f8-7a9d-4814-871b-8349744550a5"). InnerVolumeSpecName "kube-api-access-b4vwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:45.594990 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.587085 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-envoy" pod "8bd011f8-7a9d-4814-871b-8349744550a5" (UID: "8bd011f8-7a9d-4814-871b-8349744550a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.595058 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.588289 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-data" pod "8bd011f8-7a9d-4814-871b-8349744550a5" (UID: "8bd011f8-7a9d-4814-871b-8349744550a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.595133 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.588584 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd011f8-7a9d-4814-871b-8349744550a5-istiod-ca-cert" pod "8bd011f8-7a9d-4814-871b-8349744550a5" (UID: "8bd011f8-7a9d-4814-871b-8349744550a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:45.595482 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.595459 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-certs" pod "8bd011f8-7a9d-4814-871b-8349744550a5" (UID: "8bd011f8-7a9d-4814-871b-8349744550a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.598455 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.596088 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bd011f8-7a9d-4814-871b-8349744550a5-istio-podinfo" pod "8bd011f8-7a9d-4814-871b-8349744550a5" (UID: "8bd011f8-7a9d-4814-871b-8349744550a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:42:45.598455 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.596477 7618 scope.go:122] "RemoveContainer" containerID="878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767" Oct 31 17:42:45.598455 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:45.597638 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767\": container with ID starting with 878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767 not found: ID does not exist" containerID="878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767" Oct 31 17:42:45.598455 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.597675 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767"} err="failed to get container status \"878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767\": rpc error: code = NotFound desc = could not find container \"878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767\": container with ID starting with 878e22f44009cfa29c62bb0d68a10f867f6bd164102dcaaa979f8fcb1bc55767 not found: ID does not exist" Oct 31 17:42:45.598455 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.597704 7618 scope.go:122] "RemoveContainer" containerID="0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d" Oct 31 17:42:45.617420 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.612208 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpnnf\" (UniqueName: \"kubernetes.io/projected/dbf57bc3-4558-446e-96bb-05d057e94b3c-kube-api-access-dpnnf\") pod \"maas-api-revoke-sub-5fb6e677-g7bn9\" (UID: \"dbf57bc3-4558-446e-96bb-05d057e94b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:45.620881 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.620845 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-istio-token" pod "8bd011f8-7a9d-4814-871b-8349744550a5" (UID: "8bd011f8-7a9d-4814-871b-8349744550a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:45.638471 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.638416 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-socket" pod "8bd011f8-7a9d-4814-871b-8349744550a5" (UID: "8bd011f8-7a9d-4814-871b-8349744550a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.694848 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.693896 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.695110 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.695076 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4vwj\" (UniqueName: \"kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-kube-api-access-b4vwj\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.695230 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.695216 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.695359 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.695336 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.695467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.695455 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd011f8-7a9d-4814-871b-8349744550a5-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.695595 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.695566 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd011f8-7a9d-4814-871b-8349744550a5-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.695727 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.695715 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd011f8-7a9d-4814-871b-8349744550a5-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.695859 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.695845 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.695971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.695957 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd011f8-7a9d-4814-871b-8349744550a5-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.701204 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.701175 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2"] Oct 31 17:42:45.705271 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.705237 7618 scope.go:122] "RemoveContainer" containerID="0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d" Oct 31 17:42:45.705878 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:45.705851 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d\": container with ID starting with 0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d not found: ID does not exist" containerID="0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d" Oct 31 17:42:45.706022 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.705996 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d"} err="failed to get container status \"0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d\": rpc error: code = NotFound desc = could not find container \"0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d\": container with ID starting with 0978dd569bdd2275ac82a938d29a8e577b05f0424a7f83fb557892d5d0e3dd4d not found: ID does not exist" Oct 31 17:42:45.706151 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.706113 7618 scope.go:122] "RemoveContainer" containerID="75da3d280119c73a000df361198ad8877d13506b2b82195e76f1414210349bf4" Oct 31 17:42:45.712199 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.712170 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0e59ca-2w8d2"] Oct 31 17:42:45.734159 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.733698 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:42:45.745221 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.744665 7618 scope.go:122] "RemoveContainer" containerID="c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05" Oct 31 17:42:45.754482 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.754114 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5"] Oct 31 17:42:45.762082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.762058 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:45.762320 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.762303 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b78c31/premium-w5-b78c31-kserve-74d5fb78c-l85h5"] Oct 31 17:42:45.782092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.782058 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz"] Oct 31 17:42:45.788918 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.788860 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t"] Oct 31 17:42:45.806867 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.806693 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-istio-token\" (UniqueName: \"kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-istio-token\") pod \"79109729-f8dc-44d2-9366-128216bc2eb6\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " Oct 31 17:42:45.806867 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.806748 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-certs\") pod \"79109729-f8dc-44d2-9366-128216bc2eb6\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " Oct 31 17:42:45.807113 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.806883 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79109729-f8dc-44d2-9366-128216bc2eb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79109729-f8dc-44d2-9366-128216bc2eb6-istiod-ca-cert\") pod \"79109729-f8dc-44d2-9366-128216bc2eb6\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " Oct 31 17:42:45.807113 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.806916 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-kube-api-access-8ngdp\" (UniqueName: \"kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-kube-api-access-8ngdp\") pod \"79109729-f8dc-44d2-9366-128216bc2eb6\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " Oct 31 17:42:45.807113 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.806951 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-envoy\") pod \"79109729-f8dc-44d2-9366-128216bc2eb6\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " Oct 31 17:42:45.807113 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.806986 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79109729-f8dc-44d2-9366-128216bc2eb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79109729-f8dc-44d2-9366-128216bc2eb6-istio-podinfo\") pod \"79109729-f8dc-44d2-9366-128216bc2eb6\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " Oct 31 17:42:45.807329 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.807188 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-credential-socket\") pod \"79109729-f8dc-44d2-9366-128216bc2eb6\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " Oct 31 17:42:45.807329 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.807288 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-socket\") pod \"79109729-f8dc-44d2-9366-128216bc2eb6\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " Oct 31 17:42:45.807433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.807348 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-data\") pod \"79109729-f8dc-44d2-9366-128216bc2eb6\" (UID: \"79109729-f8dc-44d2-9366-128216bc2eb6\") " Oct 31 17:42:45.809635 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.808138 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-certs" pod "79109729-f8dc-44d2-9366-128216bc2eb6" (UID: "79109729-f8dc-44d2-9366-128216bc2eb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.809635 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.808151 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-data" pod "79109729-f8dc-44d2-9366-128216bc2eb6" (UID: "79109729-f8dc-44d2-9366-128216bc2eb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.809635 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.809174 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-credential-socket" pod "79109729-f8dc-44d2-9366-128216bc2eb6" (UID: "79109729-f8dc-44d2-9366-128216bc2eb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.809635 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.809427 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-socket" pod "79109729-f8dc-44d2-9366-128216bc2eb6" (UID: "79109729-f8dc-44d2-9366-128216bc2eb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.809635 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.809602 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79109729-f8dc-44d2-9366-128216bc2eb6-istiod-ca-cert" pod "79109729-f8dc-44d2-9366-128216bc2eb6" (UID: "79109729-f8dc-44d2-9366-128216bc2eb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:45.812517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.812480 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-kube-api-access-8ngdp" pod "79109729-f8dc-44d2-9366-128216bc2eb6" (UID: "79109729-f8dc-44d2-9366-128216bc2eb6"). InnerVolumeSpecName "kube-api-access-8ngdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:45.813797 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.813766 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-envoy" pod "79109729-f8dc-44d2-9366-128216bc2eb6" (UID: "79109729-f8dc-44d2-9366-128216bc2eb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:45.813866 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.813772 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-istio-token" pod "79109729-f8dc-44d2-9366-128216bc2eb6" (UID: "79109729-f8dc-44d2-9366-128216bc2eb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:45.813866 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.813807 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ee816e/distinct-2-w0-ee816e-kserve-f765968c4-9lr2t"] Oct 31 17:42:45.813866 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.813809 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79109729-f8dc-44d2-9366-128216bc2eb6-istio-podinfo" pod "79109729-f8dc-44d2-9366-128216bc2eb6" (UID: "79109729-f8dc-44d2-9366-128216bc2eb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:42:45.840834 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.840772 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr"] Oct 31 17:42:45.843308 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.843202 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0e59ca-7d585568bc-nnfxr"] Oct 31 17:42:45.850621 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.850592 7618 scope.go:122] "RemoveContainer" containerID="c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05" Oct 31 17:42:45.854060 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:45.853863 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05\": container with ID starting with c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05 not found: ID does not exist" containerID="c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05" Oct 31 17:42:45.854060 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.853919 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05"} err="failed to get container status \"c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05\": rpc error: code = NotFound desc = could not find container \"c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05\": container with ID starting with c42e1bb47d2cf7f154e4ab42eda4d88d2cec79bc5e86be9fedc410e6215bbe05 not found: ID does not exist" Oct 31 17:42:45.854060 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.853945 7618 scope.go:122] "RemoveContainer" containerID="73c514d6dac9e1b2ff0750f882859e26e9a6a3a7d6ec528d7f56b5827e8ab079" Oct 31 17:42:45.877430 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.877392 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc"] Oct 31 17:42:45.884333 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.884237 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-6gztc"] Oct 31 17:42:45.909226 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.909171 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl"] Oct 31 17:42:45.910138 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.910106 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79109729-f8dc-44d2-9366-128216bc2eb6-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.910198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.910146 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ngdp\" (UniqueName: \"kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-kube-api-access-8ngdp\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.910198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.910164 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.910198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.910177 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79109729-f8dc-44d2-9366-128216bc2eb6-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.910198 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.910190 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.910344 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.910206 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.910344 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.910222 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.910344 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.910235 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79109729-f8dc-44d2-9366-128216bc2eb6-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.910344 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.910265 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79109729-f8dc-44d2-9366-128216bc2eb6-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:45.915616 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:45.915585 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1c0bfd/simulator-w4-1c0bfd-kserve-67fc77d5f-7dxgl"] Oct 31 17:42:46.172840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.172808 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9"] Oct 31 17:42:46.175102 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:42:46.175072 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf57bc3_4558_446e_96bb_05d057e94b3c.slice/crio-82559e6e460a1bc1ef6bfd3a709eb471195f7d642b414461ca12b647b0fb0505 WatchSource:0}: Error finding container 82559e6e460a1bc1ef6bfd3a709eb471195f7d642b414461ca12b647b0fb0505: Status 404 returned error can't find the container with id 82559e6e460a1bc1ef6bfd3a709eb471195f7d642b414461ca12b647b0fb0505 Oct 31 17:42:46.376396 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.376365 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5803fb07-04c1-4d98-8278-b0f21077d21b" path="/var/lib/kubelet/pods/5803fb07-04c1-4d98-8278-b0f21077d21b/volumes" Oct 31 17:42:46.376798 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.376785 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c01da1b-48fc-49e4-9341-54965a5c45a0" path="/var/lib/kubelet/pods/8c01da1b-48fc-49e4-9341-54965a5c45a0/volumes" Oct 31 17:42:46.377302 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.377287 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08f2d28-721a-4f5c-9e8e-db5aefafdd0a" path="/var/lib/kubelet/pods/a08f2d28-721a-4f5c-9e8e-db5aefafdd0a/volumes" Oct 31 17:42:46.378105 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.378047 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f22d3d-03f9-495b-9b3e-bbc3aea680fd" path="/var/lib/kubelet/pods/b4f22d3d-03f9-495b-9b3e-bbc3aea680fd/volumes" Oct 31 17:42:46.378513 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.378499 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ab7228-ad0c-4a75-b452-550e9db7bc8b" path="/var/lib/kubelet/pods/e1ab7228-ad0c-4a75-b452-550e9db7bc8b/volumes" Oct 31 17:42:46.378968 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.378952 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c13943-5746-4c6e-a4d2-5c11e89156bd" path="/var/lib/kubelet/pods/f7c13943-5746-4c6e-a4d2-5c11e89156bd/volumes" Oct 31 17:42:46.571407 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.571380 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" Oct 31 17:42:46.571589 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.571444 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg" event={"ID":"8bd011f8-7a9d-4814-871b-8349744550a5","Type":"ContainerDied","Data":"afa71151a330d2e4f0c7a3671607236e20e1763cfb61b66c6590a485743cfb15"} Oct 31 17:42:46.575667 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.574703 7618 generic.go:356] "Generic (PLEG): container finished" podID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerID="8387da5b989ca9d565255372597055dbe84b41509cf4ad5935c7898570e69b8e" exitCode=0 Oct 31 17:42:46.575667 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.574817 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" event={"ID":"4b7b20a8-644d-45fb-8055-4f0a8a90c48d","Type":"ContainerDied","Data":"8387da5b989ca9d565255372597055dbe84b41509cf4ad5935c7898570e69b8e"} Oct 31 17:42:46.577930 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.577742 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" Oct 31 17:42:46.577930 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.577775 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv" event={"ID":"79109729-f8dc-44d2-9366-128216bc2eb6","Type":"ContainerDied","Data":"4432772a9fcebe27041ee5d47cb86f2f3788afd6a099554e449d44f154abdd4c"} Oct 31 17:42:46.577930 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.577839 7618 scope.go:122] "RemoveContainer" containerID="e3bd2e57f25fd5c2681fb5e95ac088666da48ea6424acfdd1f556476d7b4bbbd" Oct 31 17:42:46.591321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.589892 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" event={"ID":"dbf57bc3-4558-446e-96bb-05d057e94b3c","Type":"ContainerStarted","Data":"82559e6e460a1bc1ef6bfd3a709eb471195f7d642b414461ca12b647b0fb0505"} Oct 31 17:42:46.592350 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.591679 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" event={"ID":"9d3288d2-a16e-4770-9e3e-b06584d07dd8","Type":"ContainerStarted","Data":"fa0b5f048055a0723ca29348269ba122d343b6337d929ff8d98b7b04ed71a6df"} Oct 31 17:42:46.592350 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.591717 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" event={"ID":"9d3288d2-a16e-4770-9e3e-b06584d07dd8","Type":"ContainerStarted","Data":"87399e65a9198b11ef79a9f11bed77e78abe72efad7998fa7c8ae58d633acce3"} Oct 31 17:42:46.595403 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.594415 7618 generic.go:356] "Generic (PLEG): container finished" podID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerID="25b33c0ea918c5c296923cb846b4001154c30b29bff7679599c59150e4ad6f70" exitCode=0 Oct 31 17:42:46.595403 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.594488 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" event={"ID":"5cf88942-00cd-41b1-9a86-9591b5e41283","Type":"ContainerDied","Data":"25b33c0ea918c5c296923cb846b4001154c30b29bff7679599c59150e4ad6f70"} Oct 31 17:42:46.667226 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.667166 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg"] Oct 31 17:42:46.674108 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.674051 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-gszxg"] Oct 31 17:42:46.682934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.682859 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv"] Oct 31 17:42:46.691961 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.691903 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cf0a86-openshift-default-595bfdc58c-vgvwv"] Oct 31 17:42:46.784586 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.784558 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:42:46.839062 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.838722 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" podStartSLOduration=2.838700766 podStartE2EDuration="2.838700766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:42:46.745589669 +0000 UTC m=+888.493564137" watchObservedRunningTime="2026-10-31 17:42:46.838700766 +0000 UTC m=+888.586675234" Oct 31 17:42:46.839572 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.839544 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf88942-00cd-41b1-9a86-9591b5e41283-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf88942-00cd-41b1-9a86-9591b5e41283-istiod-ca-cert\") pod \"5cf88942-00cd-41b1-9a86-9591b5e41283\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " Oct 31 17:42:46.839705 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.839685 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-envoy\") pod \"5cf88942-00cd-41b1-9a86-9591b5e41283\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " Oct 31 17:42:46.839785 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.839767 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-kube-api-access-w27kd\" (UniqueName: \"kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-kube-api-access-w27kd\") pod \"5cf88942-00cd-41b1-9a86-9591b5e41283\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " Oct 31 17:42:46.839850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.839830 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-podinfo\") pod \"5cf88942-00cd-41b1-9a86-9591b5e41283\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " Oct 31 17:42:46.839906 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.839876 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-credential-socket\") pod \"5cf88942-00cd-41b1-9a86-9591b5e41283\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " Oct 31 17:42:46.839962 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.839919 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-certs\") pod \"5cf88942-00cd-41b1-9a86-9591b5e41283\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " Oct 31 17:42:46.840020 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.839975 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-data\") pod \"5cf88942-00cd-41b1-9a86-9591b5e41283\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " Oct 31 17:42:46.840020 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.840009 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-token\" (UniqueName: \"kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-token\") pod \"5cf88942-00cd-41b1-9a86-9591b5e41283\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " Oct 31 17:42:46.840128 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.840113 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-socket\") pod \"5cf88942-00cd-41b1-9a86-9591b5e41283\" (UID: \"5cf88942-00cd-41b1-9a86-9591b5e41283\") " Oct 31 17:42:46.841038 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.840998 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-socket" pod "5cf88942-00cd-41b1-9a86-9591b5e41283" (UID: "5cf88942-00cd-41b1-9a86-9591b5e41283"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:46.841232 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.841205 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-credential-socket" pod "5cf88942-00cd-41b1-9a86-9591b5e41283" (UID: "5cf88942-00cd-41b1-9a86-9591b5e41283"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:46.841374 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.841214 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-certs" pod "5cf88942-00cd-41b1-9a86-9591b5e41283" (UID: "5cf88942-00cd-41b1-9a86-9591b5e41283"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:46.841601 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.841579 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-data" pod "5cf88942-00cd-41b1-9a86-9591b5e41283" (UID: "5cf88942-00cd-41b1-9a86-9591b5e41283"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:46.842717 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.842689 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf88942-00cd-41b1-9a86-9591b5e41283-istiod-ca-cert" pod "5cf88942-00cd-41b1-9a86-9591b5e41283" (UID: "5cf88942-00cd-41b1-9a86-9591b5e41283"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:46.845118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.844462 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-token" pod "5cf88942-00cd-41b1-9a86-9591b5e41283" (UID: "5cf88942-00cd-41b1-9a86-9591b5e41283"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:46.845118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.844539 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-kube-api-access-w27kd" pod "5cf88942-00cd-41b1-9a86-9591b5e41283" (UID: "5cf88942-00cd-41b1-9a86-9591b5e41283"). InnerVolumeSpecName "kube-api-access-w27kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:46.845118 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.844594 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-podinfo" pod "5cf88942-00cd-41b1-9a86-9591b5e41283" (UID: "5cf88942-00cd-41b1-9a86-9591b5e41283"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:42:46.846701 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.846674 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-envoy" pod "5cf88942-00cd-41b1-9a86-9591b5e41283" (UID: "5cf88942-00cd-41b1-9a86-9591b5e41283"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:46.877404 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.877190 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:42:46.883516 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.883489 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:46.941757 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.941511 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-podinfo\") pod \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " Oct 31 17:42:46.941757 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.941620 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-socket\") pod \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " Oct 31 17:42:46.941757 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.941677 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-kube-api-access-58cl8\" (UniqueName: \"kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-kube-api-access-58cl8\") pod \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " Oct 31 17:42:46.941757 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.941713 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-token\" (UniqueName: \"kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-token\") pod \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " Oct 31 17:42:46.941757 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.941752 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-xks-ca\") pod \"c586ef82-a0d9-4dcb-8504-339d26c88b04\" (UID: \"c586ef82-a0d9-4dcb-8504-339d26c88b04\") " Oct 31 17:42:46.942129 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.941781 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-openshift-ca\") pod \"c586ef82-a0d9-4dcb-8504-339d26c88b04\" (UID: \"c586ef82-a0d9-4dcb-8504-339d26c88b04\") " Oct 31 17:42:46.942129 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.941889 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istiod-ca-cert\") pod \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " Oct 31 17:42:46.942129 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.941916 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-data\") pod \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " Oct 31 17:42:46.942129 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.941947 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c586ef82-a0d9-4dcb-8504-339d26c88b04-kube-api-access-xnxgd\" (UniqueName: \"kubernetes.io/projected/c586ef82-a0d9-4dcb-8504-339d26c88b04-kube-api-access-xnxgd\") pod \"c586ef82-a0d9-4dcb-8504-339d26c88b04\" (UID: \"c586ef82-a0d9-4dcb-8504-339d26c88b04\") " Oct 31 17:42:46.942129 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.942091 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-credential-socket\") pod \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " Oct 31 17:42:46.942421 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.942141 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-certs\") pod \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " Oct 31 17:42:46.942421 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.942236 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-envoy\") pod \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\" (UID: \"4b7b20a8-644d-45fb-8055-4f0a8a90c48d\") " Oct 31 17:42:46.942964 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.942938 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:46.943075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.942971 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w27kd\" (UniqueName: \"kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-kube-api-access-w27kd\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:46.943075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.942987 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:46.943075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.943004 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:46.943075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.943018 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:46.943075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.943033 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:46.943075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.943044 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf88942-00cd-41b1-9a86-9591b5e41283-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:46.943075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.943055 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf88942-00cd-41b1-9a86-9591b5e41283-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:46.943075 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.943068 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf88942-00cd-41b1-9a86-9591b5e41283-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:46.943991 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.943964 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-openshift-ca" pod "c586ef82-a0d9-4dcb-8504-339d26c88b04" (UID: "c586ef82-a0d9-4dcb-8504-339d26c88b04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:46.945836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.945521 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-credential-socket" pod "4b7b20a8-644d-45fb-8055-4f0a8a90c48d" (UID: "4b7b20a8-644d-45fb-8055-4f0a8a90c48d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:46.945836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.945804 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-certs" pod "4b7b20a8-644d-45fb-8055-4f0a8a90c48d" (UID: "4b7b20a8-644d-45fb-8055-4f0a8a90c48d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:46.946038 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.946001 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-envoy" pod "4b7b20a8-644d-45fb-8055-4f0a8a90c48d" (UID: "4b7b20a8-644d-45fb-8055-4f0a8a90c48d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:46.946151 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.946030 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-socket" pod "4b7b20a8-644d-45fb-8055-4f0a8a90c48d" (UID: "4b7b20a8-644d-45fb-8055-4f0a8a90c48d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:46.946341 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.946213 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istiod-ca-cert" pod "4b7b20a8-644d-45fb-8055-4f0a8a90c48d" (UID: "4b7b20a8-644d-45fb-8055-4f0a8a90c48d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:46.946341 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.946306 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-data" pod "4b7b20a8-644d-45fb-8055-4f0a8a90c48d" (UID: "4b7b20a8-644d-45fb-8055-4f0a8a90c48d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:46.968817 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.967870 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75"] Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969653 7618 cpu_manager.go:414] "containerMap: removing container" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" containerName="istio-proxy" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969681 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" containerName="istio-proxy" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969716 7618 cpu_manager.go:414] "containerMap: removing container" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969724 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969747 7618 cpu_manager.go:414] "containerMap: removing container" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969755 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969801 7618 cpu_manager.go:414] "containerMap: removing container" podUID="c586ef82-a0d9-4dcb-8504-339d26c88b04" containerName="revoke-keys" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969809 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c586ef82-a0d9-4dcb-8504-339d26c88b04" containerName="revoke-keys" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969829 7618 cpu_manager.go:414] "containerMap: removing container" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969836 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969959 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" containerName="istio-proxy" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969975 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" containerName="istio-proxy" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.969990 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="c586ef82-a0d9-4dcb-8504-339d26c88b04" containerName="revoke-keys" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.970004 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" containerName="istio-proxy" Oct 31 17:42:46.970938 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.970015 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" containerName="istio-proxy" Oct 31 17:42:46.973693 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.973635 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-token" pod "4b7b20a8-644d-45fb-8055-4f0a8a90c48d" (UID: "4b7b20a8-644d-45fb-8055-4f0a8a90c48d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:46.974546 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.974517 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-kube-api-access-58cl8" pod "4b7b20a8-644d-45fb-8055-4f0a8a90c48d" (UID: "4b7b20a8-644d-45fb-8055-4f0a8a90c48d"). InnerVolumeSpecName "kube-api-access-58cl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:46.980494 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.977623 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:46.980494 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.978493 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-xks-ca" pod "c586ef82-a0d9-4dcb-8504-339d26c88b04" (UID: "c586ef82-a0d9-4dcb-8504-339d26c88b04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:46.980494 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.978590 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c586ef82-a0d9-4dcb-8504-339d26c88b04-kube-api-access-xnxgd" pod "c586ef82-a0d9-4dcb-8504-339d26c88b04" (UID: "c586ef82-a0d9-4dcb-8504-339d26c88b04"). InnerVolumeSpecName "kube-api-access-xnxgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:46.980494 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.980444 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-podinfo" pod "4b7b20a8-644d-45fb-8055-4f0a8a90c48d" (UID: "4b7b20a8-644d-45fb-8055-4f0a8a90c48d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:42:46.989008 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:46.987407 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75"] Oct 31 17:42:47.045294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.044654 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-credential-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.045294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.044711 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istiod-ca-cert\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.045294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.044745 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-token\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.045294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.044850 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-podinfo\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.045294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.044895 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hts67\" (UniqueName: \"kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-kube-api-access-hts67\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.045294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.044957 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-data\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.045294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.044990 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-envoy\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.045294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.045070 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-certs\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.045294 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.045097 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.046327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.045246 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.046327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.045955 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.046327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.045972 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.046327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.045999 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58cl8\" (UniqueName: \"kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-kube-api-access-58cl8\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.046327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.046014 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.046327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.046030 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.046327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.046044 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c586ef82-a0d9-4dcb-8504-339d26c88b04-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.046327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.046057 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.046327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.046071 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.046327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.046085 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnxgd\" (UniqueName: \"kubernetes.io/projected/c586ef82-a0d9-4dcb-8504-339d26c88b04-kube-api-access-xnxgd\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.046327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.046099 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.046327 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.046113 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7b20a8-644d-45fb-8055-4f0a8a90c48d-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.147836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.147025 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-certs\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.147836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.147071 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.147836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.147183 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-credential-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.147836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.147207 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istiod-ca-cert\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.147836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.147228 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-token\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.147836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.147311 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-podinfo\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.147836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.147336 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hts67\" (UniqueName: \"kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-kube-api-access-hts67\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.147836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.147378 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-data\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.147836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.147397 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-envoy\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.148772 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.148526 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-certs\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.148840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.148816 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.149321 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.149056 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-credential-socket\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.149631 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.149589 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-data\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.149910 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.149878 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istiod-ca-cert\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.150460 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.150410 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-envoy\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.152639 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.152616 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-podinfo\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.165828 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.165798 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:42:47.178034 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.178007 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-token\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.178223 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.178202 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hts67\" (UniqueName: \"kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-kube-api-access-hts67\") pod \"e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.248603 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.248510 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-socket\") pod \"bac3a54c-5d8f-4451-9584-dec64d6d235c\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " Oct 31 17:42:47.248603 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.248559 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-certs\") pod \"bac3a54c-5d8f-4451-9584-dec64d6d235c\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " Oct 31 17:42:47.248868 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.248647 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-token\" (UniqueName: \"kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-token\") pod \"bac3a54c-5d8f-4451-9584-dec64d6d235c\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " Oct 31 17:42:47.248868 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.248745 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-kube-api-access-vpslm\" (UniqueName: \"kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-kube-api-access-vpslm\") pod \"bac3a54c-5d8f-4451-9584-dec64d6d235c\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " Oct 31 17:42:47.248868 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.248812 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac3a54c-5d8f-4451-9584-dec64d6d235c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac3a54c-5d8f-4451-9584-dec64d6d235c-istiod-ca-cert\") pod \"bac3a54c-5d8f-4451-9584-dec64d6d235c\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " Oct 31 17:42:47.249047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.248963 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-podinfo\") pod \"bac3a54c-5d8f-4451-9584-dec64d6d235c\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " Oct 31 17:42:47.249047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.249014 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-envoy\") pod \"bac3a54c-5d8f-4451-9584-dec64d6d235c\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " Oct 31 17:42:47.249153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.249068 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-credential-socket\") pod \"bac3a54c-5d8f-4451-9584-dec64d6d235c\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " Oct 31 17:42:47.249153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.249109 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-data\") pod \"bac3a54c-5d8f-4451-9584-dec64d6d235c\" (UID: \"bac3a54c-5d8f-4451-9584-dec64d6d235c\") " Oct 31 17:42:47.249910 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.249883 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-data" pod "bac3a54c-5d8f-4451-9584-dec64d6d235c" (UID: "bac3a54c-5d8f-4451-9584-dec64d6d235c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:47.250282 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.250243 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-credential-socket" pod "bac3a54c-5d8f-4451-9584-dec64d6d235c" (UID: "bac3a54c-5d8f-4451-9584-dec64d6d235c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:47.250682 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.250663 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac3a54c-5d8f-4451-9584-dec64d6d235c-istiod-ca-cert" pod "bac3a54c-5d8f-4451-9584-dec64d6d235c" (UID: "bac3a54c-5d8f-4451-9584-dec64d6d235c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:47.250863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.250842 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-socket" pod "bac3a54c-5d8f-4451-9584-dec64d6d235c" (UID: "bac3a54c-5d8f-4451-9584-dec64d6d235c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:47.251030 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.251008 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-certs" pod "bac3a54c-5d8f-4451-9584-dec64d6d235c" (UID: "bac3a54c-5d8f-4451-9584-dec64d6d235c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:47.254643 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.254619 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-kube-api-access-vpslm" pod "bac3a54c-5d8f-4451-9584-dec64d6d235c" (UID: "bac3a54c-5d8f-4451-9584-dec64d6d235c"). InnerVolumeSpecName "kube-api-access-vpslm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:47.254765 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.254747 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-token" pod "bac3a54c-5d8f-4451-9584-dec64d6d235c" (UID: "bac3a54c-5d8f-4451-9584-dec64d6d235c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:47.254868 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.254619 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-podinfo" pod "bac3a54c-5d8f-4451-9584-dec64d6d235c" (UID: "bac3a54c-5d8f-4451-9584-dec64d6d235c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:42:47.256523 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.256493 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-envoy" pod "bac3a54c-5d8f-4451-9584-dec64d6d235c" (UID: "bac3a54c-5d8f-4451-9584-dec64d6d235c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:47.315708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.315662 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:47.351941 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.351899 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.351941 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.351944 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpslm\" (UniqueName: \"kubernetes.io/projected/bac3a54c-5d8f-4451-9584-dec64d6d235c-kube-api-access-vpslm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.352180 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.351963 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac3a54c-5d8f-4451-9584-dec64d6d235c-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.352180 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.351981 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.352180 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.351997 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.352180 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.352013 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.352180 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.352029 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.352180 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.352045 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.352180 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.352063 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac3a54c-5d8f-4451-9584-dec64d6d235c-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:47.608309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.608271 7618 generic.go:356] "Generic (PLEG): container finished" podID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerID="cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1" exitCode=0 Oct 31 17:42:47.608798 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.608315 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" event={"ID":"bac3a54c-5d8f-4451-9584-dec64d6d235c","Type":"ContainerDied","Data":"cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1"} Oct 31 17:42:47.608798 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.608348 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" Oct 31 17:42:47.608798 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.608385 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v" event={"ID":"bac3a54c-5d8f-4451-9584-dec64d6d235c","Type":"ContainerDied","Data":"928cc4ef305590ed91c20928abeec94294b36c50ee880e166c04288ebd4e038c"} Oct 31 17:42:47.608798 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.608418 7618 scope.go:122] "RemoveContainer" containerID="cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1" Oct 31 17:42:47.611331 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.610078 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" Oct 31 17:42:47.611331 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.610104 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5" event={"ID":"4b7b20a8-644d-45fb-8055-4f0a8a90c48d","Type":"ContainerDied","Data":"01314b38020e463d9e1cd718bf5949ccb69140d5b67fb699b9afe0b4faee0fb3"} Oct 31 17:42:47.613238 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.613212 7618 generic.go:356] "Generic (PLEG): container finished" podID="dbf57bc3-4558-446e-96bb-05d057e94b3c" containerID="49ca131219313165d7f092acf44dd40ff01b2c7b36f86950419b6f4769b9fb94" exitCode=0 Oct 31 17:42:47.613431 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.613282 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" event={"ID":"dbf57bc3-4558-446e-96bb-05d057e94b3c","Type":"ContainerDied","Data":"49ca131219313165d7f092acf44dd40ff01b2c7b36f86950419b6f4769b9fb94"} Oct 31 17:42:47.616609 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.616497 7618 generic.go:356] "Generic (PLEG): container finished" podID="9d3288d2-a16e-4770-9e3e-b06584d07dd8" containerID="fa0b5f048055a0723ca29348269ba122d343b6337d929ff8d98b7b04ed71a6df" exitCode=0 Oct 31 17:42:47.617314 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.616589 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" event={"ID":"9d3288d2-a16e-4770-9e3e-b06584d07dd8","Type":"ContainerDied","Data":"fa0b5f048055a0723ca29348269ba122d343b6337d929ff8d98b7b04ed71a6df"} Oct 31 17:42:47.618522 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.618501 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" Oct 31 17:42:47.618626 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.618558 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht" event={"ID":"5cf88942-00cd-41b1-9a86-9591b5e41283","Type":"ContainerDied","Data":"5423315ad5e7642074baf7b0f15f78c0dc90f51bd1620d1e21a458d4d5937ae2"} Oct 31 17:42:47.620146 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.620118 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" event={"ID":"c586ef82-a0d9-4dcb-8504-339d26c88b04","Type":"ContainerDied","Data":"7beba9d458a3c923cb2c330339106cf5bf75741c83e48c32e01de9174b69767f"} Oct 31 17:42:47.620236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.620167 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7beba9d458a3c923cb2c330339106cf5bf75741c83e48c32e01de9174b69767f" Oct 31 17:42:47.620351 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.620241 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk" Oct 31 17:42:47.632163 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.632141 7618 scope.go:122] "RemoveContainer" containerID="cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1" Oct 31 17:42:47.632575 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:47.632547 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1\": container with ID starting with cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1 not found: ID does not exist" containerID="cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1" Oct 31 17:42:47.632671 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.632590 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1"} err="failed to get container status \"cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1\": rpc error: code = NotFound desc = could not find container \"cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1\": container with ID starting with cd206be0e74a589c206ab15d7e1eedcaee60cd9e6a198fb48906ee6f6fdc9ea1 not found: ID does not exist" Oct 31 17:42:47.632671 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.632618 7618 scope.go:122] "RemoveContainer" containerID="8387da5b989ca9d565255372597055dbe84b41509cf4ad5935c7898570e69b8e" Oct 31 17:42:47.651340 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.650498 7618 scope.go:122] "RemoveContainer" containerID="25b33c0ea918c5c296923cb846b4001154c30b29bff7679599c59150e4ad6f70" Oct 31 17:42:47.655011 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.654982 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75"] Oct 31 17:42:47.656323 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.656286 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:42:47.656440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.656365 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:42:47.656440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.656409 7618 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 17:42:47.763550 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.763520 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht"] Oct 31 17:42:47.771199 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.771165 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-dvxht"] Oct 31 17:42:47.815012 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.814886 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5"] Oct 31 17:42:47.821380 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.821343 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-b6rp5"] Oct 31 17:42:47.849796 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.849730 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v"] Oct 31 17:42:47.853597 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:47.853564 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c446c-openshift-default-797c6dc855-hk56v"] Oct 31 17:42:48.376004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:48.375970 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b7b20a8-644d-45fb-8055-4f0a8a90c48d" path="/var/lib/kubelet/pods/4b7b20a8-644d-45fb-8055-4f0a8a90c48d/volumes" Oct 31 17:42:48.376532 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:48.376514 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf88942-00cd-41b1-9a86-9591b5e41283" path="/var/lib/kubelet/pods/5cf88942-00cd-41b1-9a86-9591b5e41283/volumes" Oct 31 17:42:48.376971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:48.376957 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79109729-f8dc-44d2-9366-128216bc2eb6" path="/var/lib/kubelet/pods/79109729-f8dc-44d2-9366-128216bc2eb6/volumes" Oct 31 17:42:48.377835 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:48.377816 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd011f8-7a9d-4814-871b-8349744550a5" path="/var/lib/kubelet/pods/8bd011f8-7a9d-4814-871b-8349744550a5/volumes" Oct 31 17:42:48.378291 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:48.378244 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" path="/var/lib/kubelet/pods/bac3a54c-5d8f-4451-9584-dec64d6d235c/volumes" Oct 31 17:42:48.668182 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:48.668135 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" event={"ID":"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7","Type":"ContainerStarted","Data":"7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf"} Oct 31 17:42:48.668592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:48.668195 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" event={"ID":"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7","Type":"ContainerStarted","Data":"56765bd84f280d686232fcbcdc60d0fe32f9cdca97f39d1c48c0cd4cfb018592"} Oct 31 17:42:48.742288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:48.741721 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" podStartSLOduration=2.741696504 podStartE2EDuration="2.741696504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:42:48.719752285 +0000 UTC m=+890.467726752" watchObservedRunningTime="2026-10-31 17:42:48.741696504 +0000 UTC m=+890.489670971" Oct 31 17:42:48.927322 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:48.925279 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:49.054785 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.054736 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:49.082707 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.082670 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-xks-ca\") pod \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\" (UID: \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\") " Oct 31 17:42:49.082914 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.082861 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3288d2-a16e-4770-9e3e-b06584d07dd8-kube-api-access-4f5sw\" (UniqueName: \"kubernetes.io/projected/9d3288d2-a16e-4770-9e3e-b06584d07dd8-kube-api-access-4f5sw\") pod \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\" (UID: \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\") " Oct 31 17:42:49.083038 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.083013 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-openshift-ca\") pod \"dbf57bc3-4558-446e-96bb-05d057e94b3c\" (UID: \"dbf57bc3-4558-446e-96bb-05d057e94b3c\") " Oct 31 17:42:49.083112 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.083096 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-openshift-ca\") pod \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\" (UID: \"9d3288d2-a16e-4770-9e3e-b06584d07dd8\") " Oct 31 17:42:49.083168 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.083121 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf57bc3-4558-446e-96bb-05d057e94b3c-kube-api-access-dpnnf\" (UniqueName: \"kubernetes.io/projected/dbf57bc3-4558-446e-96bb-05d057e94b3c-kube-api-access-dpnnf\") pod \"dbf57bc3-4558-446e-96bb-05d057e94b3c\" (UID: \"dbf57bc3-4558-446e-96bb-05d057e94b3c\") " Oct 31 17:42:49.084001 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.083612 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-openshift-ca" pod "dbf57bc3-4558-446e-96bb-05d057e94b3c" (UID: "dbf57bc3-4558-446e-96bb-05d057e94b3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:49.084001 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.083908 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-openshift-ca" pod "9d3288d2-a16e-4770-9e3e-b06584d07dd8" (UID: "9d3288d2-a16e-4770-9e3e-b06584d07dd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:49.086128 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.086103 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf57bc3-4558-446e-96bb-05d057e94b3c-kube-api-access-dpnnf" pod "dbf57bc3-4558-446e-96bb-05d057e94b3c" (UID: "dbf57bc3-4558-446e-96bb-05d057e94b3c"). InnerVolumeSpecName "kube-api-access-dpnnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:49.086226 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.086141 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3288d2-a16e-4770-9e3e-b06584d07dd8-kube-api-access-4f5sw" pod "9d3288d2-a16e-4770-9e3e-b06584d07dd8" (UID: "9d3288d2-a16e-4770-9e3e-b06584d07dd8"). InnerVolumeSpecName "kube-api-access-4f5sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:49.086226 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.086169 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-xks-ca" pod "9d3288d2-a16e-4770-9e3e-b06584d07dd8" (UID: "9d3288d2-a16e-4770-9e3e-b06584d07dd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:49.184309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.184222 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-xks-ca\") pod \"dbf57bc3-4558-446e-96bb-05d057e94b3c\" (UID: \"dbf57bc3-4558-446e-96bb-05d057e94b3c\") " Oct 31 17:42:49.184632 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.184616 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:49.184676 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.184637 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:49.184676 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.184646 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpnnf\" (UniqueName: \"kubernetes.io/projected/dbf57bc3-4558-446e-96bb-05d057e94b3c-kube-api-access-dpnnf\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:49.184676 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.184655 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3288d2-a16e-4770-9e3e-b06584d07dd8-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:49.184676 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.184664 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f5sw\" (UniqueName: \"kubernetes.io/projected/9d3288d2-a16e-4770-9e3e-b06584d07dd8-kube-api-access-4f5sw\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:49.186261 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.186229 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-xks-ca" pod "dbf57bc3-4558-446e-96bb-05d057e94b3c" (UID: "dbf57bc3-4558-446e-96bb-05d057e94b3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:49.288875 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.285917 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbf57bc3-4558-446e-96bb-05d057e94b3c-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:49.320848 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.320810 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:49.336103 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.336064 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 17:42:49.336395 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.336359 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" podUID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 17:42:49.682203 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.682175 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" Oct 31 17:42:49.685328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.685301 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9" event={"ID":"dbf57bc3-4558-446e-96bb-05d057e94b3c","Type":"ContainerDied","Data":"82559e6e460a1bc1ef6bfd3a709eb471195f7d642b414461ca12b647b0fb0505"} Oct 31 17:42:49.685485 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.685473 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82559e6e460a1bc1ef6bfd3a709eb471195f7d642b414461ca12b647b0fb0505" Oct 31 17:42:49.685565 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.685555 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" event={"ID":"9d3288d2-a16e-4770-9e3e-b06584d07dd8","Type":"ContainerDied","Data":"87399e65a9198b11ef79a9f11bed77e78abe72efad7998fa7c8ae58d633acce3"} Oct 31 17:42:49.685628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.685620 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87399e65a9198b11ef79a9f11bed77e78abe72efad7998fa7c8ae58d633acce3" Oct 31 17:42:49.685680 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:49.685482 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz" Oct 31 17:42:50.319708 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:50.319664 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 17:42:50.319991 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:50.319948 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" podUID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 17:42:50.813691 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:50.813653 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" containerID="cri-o://c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3" gracePeriod=45 Oct 31 17:42:50.957709 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:50.957664 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" containerID="cri-o://c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c" gracePeriod=45 Oct 31 17:42:51.101829 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.101731 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": dial tcp 10.134.0.74:8000: connect: connection refused" Oct 31 17:42:51.210743 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.210636 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:42:51.314411 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.314379 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-dshm\") pod \"67be76a3-2fc6-4e3e-8690-7f6549760cea\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " Oct 31 17:42:51.314762 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.314736 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-tmp-dir\") pod \"67be76a3-2fc6-4e3e-8690-7f6549760cea\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " Oct 31 17:42:51.314917 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.314904 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-home\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-home\") pod \"67be76a3-2fc6-4e3e-8690-7f6549760cea\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " Oct 31 17:42:51.315023 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.315012 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67be76a3-2fc6-4e3e-8690-7f6549760cea-kube-api-access-nrfbk\" (UniqueName: \"kubernetes.io/projected/67be76a3-2fc6-4e3e-8690-7f6549760cea-kube-api-access-nrfbk\") pod \"67be76a3-2fc6-4e3e-8690-7f6549760cea\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " Oct 31 17:42:51.315108 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.315097 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67be76a3-2fc6-4e3e-8690-7f6549760cea-tls-certs\" (UniqueName: \"kubernetes.io/secret/67be76a3-2fc6-4e3e-8690-7f6549760cea-tls-certs\") pod \"67be76a3-2fc6-4e3e-8690-7f6549760cea\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " Oct 31 17:42:51.342334 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.315390 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-model-cache\") pod \"67be76a3-2fc6-4e3e-8690-7f6549760cea\" (UID: \"67be76a3-2fc6-4e3e-8690-7f6549760cea\") " Oct 31 17:42:51.354146 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.349389 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 17:42:51.354146 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.349460 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" podUID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 17:42:51.354627 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.354600 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-model-cache" pod "67be76a3-2fc6-4e3e-8690-7f6549760cea" (UID: "67be76a3-2fc6-4e3e-8690-7f6549760cea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:51.355146 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.355111 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-tmp-dir" pod "67be76a3-2fc6-4e3e-8690-7f6549760cea" (UID: "67be76a3-2fc6-4e3e-8690-7f6549760cea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:51.358604 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.358560 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-dshm" pod "67be76a3-2fc6-4e3e-8690-7f6549760cea" (UID: "67be76a3-2fc6-4e3e-8690-7f6549760cea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:51.358604 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.358560 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67be76a3-2fc6-4e3e-8690-7f6549760cea-tls-certs" pod "67be76a3-2fc6-4e3e-8690-7f6549760cea" (UID: "67be76a3-2fc6-4e3e-8690-7f6549760cea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:51.358717 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.358606 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67be76a3-2fc6-4e3e-8690-7f6549760cea-kube-api-access-nrfbk" pod "67be76a3-2fc6-4e3e-8690-7f6549760cea" (UID: "67be76a3-2fc6-4e3e-8690-7f6549760cea"). InnerVolumeSpecName "kube-api-access-nrfbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:51.359236 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.359214 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-home" pod "67be76a3-2fc6-4e3e-8690-7f6549760cea" (UID: "67be76a3-2fc6-4e3e-8690-7f6549760cea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:51.382915 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.382888 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:42:51.457291 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.451626 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-home\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-home\") pod \"58baa22b-b233-4796-82d2-983a7ec80b0b\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " Oct 31 17:42:51.457291 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.451737 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-tmp-dir\") pod \"58baa22b-b233-4796-82d2-983a7ec80b0b\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " Oct 31 17:42:51.457291 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.451793 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-model-cache\") pod \"58baa22b-b233-4796-82d2-983a7ec80b0b\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " Oct 31 17:42:51.457291 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.451844 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58baa22b-b233-4796-82d2-983a7ec80b0b-kube-api-access-zfvlh\" (UniqueName: \"kubernetes.io/projected/58baa22b-b233-4796-82d2-983a7ec80b0b-kube-api-access-zfvlh\") pod \"58baa22b-b233-4796-82d2-983a7ec80b0b\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " Oct 31 17:42:51.457291 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.452052 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-dshm\") pod \"58baa22b-b233-4796-82d2-983a7ec80b0b\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " Oct 31 17:42:51.457291 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.452083 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58baa22b-b233-4796-82d2-983a7ec80b0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/58baa22b-b233-4796-82d2-983a7ec80b0b-tls-certs\") pod \"58baa22b-b233-4796-82d2-983a7ec80b0b\" (UID: \"58baa22b-b233-4796-82d2-983a7ec80b0b\") " Oct 31 17:42:51.459900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.458697 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:51.459900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.458733 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:51.459900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.458753 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrfbk\" (UniqueName: \"kubernetes.io/projected/67be76a3-2fc6-4e3e-8690-7f6549760cea-kube-api-access-nrfbk\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:51.459900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.458767 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67be76a3-2fc6-4e3e-8690-7f6549760cea-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:51.459900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.458782 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:51.459900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.458795 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67be76a3-2fc6-4e3e-8690-7f6549760cea-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:51.460585 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.460549 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-model-cache" pod "58baa22b-b233-4796-82d2-983a7ec80b0b" (UID: "58baa22b-b233-4796-82d2-983a7ec80b0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:51.460839 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.460818 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-home" pod "58baa22b-b233-4796-82d2-983a7ec80b0b" (UID: "58baa22b-b233-4796-82d2-983a7ec80b0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:51.461032 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.461006 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-tmp-dir" pod "58baa22b-b233-4796-82d2-983a7ec80b0b" (UID: "58baa22b-b233-4796-82d2-983a7ec80b0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:51.461592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.461563 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58baa22b-b233-4796-82d2-983a7ec80b0b-tls-certs" pod "58baa22b-b233-4796-82d2-983a7ec80b0b" (UID: "58baa22b-b233-4796-82d2-983a7ec80b0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:51.463374 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.463338 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58baa22b-b233-4796-82d2-983a7ec80b0b-kube-api-access-zfvlh" pod "58baa22b-b233-4796-82d2-983a7ec80b0b" (UID: "58baa22b-b233-4796-82d2-983a7ec80b0b"). InnerVolumeSpecName "kube-api-access-zfvlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:51.463476 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.463370 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-dshm" pod "58baa22b-b233-4796-82d2-983a7ec80b0b" (UID: "58baa22b-b233-4796-82d2-983a7ec80b0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:51.560378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.560292 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:51.560378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.560319 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:51.560378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.560330 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:51.560378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.560339 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfvlh\" (UniqueName: \"kubernetes.io/projected/58baa22b-b233-4796-82d2-983a7ec80b0b-kube-api-access-zfvlh\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:51.560378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.560348 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58baa22b-b233-4796-82d2-983a7ec80b0b-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:51.560378 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.560356 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58baa22b-b233-4796-82d2-983a7ec80b0b-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:51.702759 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.702724 7618 generic.go:356] "Generic (PLEG): container finished" podID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerID="c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3" exitCode=0 Oct 31 17:42:51.702951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.702775 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" Oct 31 17:42:51.702951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.702821 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" event={"ID":"67be76a3-2fc6-4e3e-8690-7f6549760cea","Type":"ContainerDied","Data":"c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3"} Oct 31 17:42:51.702951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.702875 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q" event={"ID":"67be76a3-2fc6-4e3e-8690-7f6549760cea","Type":"ContainerDied","Data":"8a4071e76d5b31e20a533f68442b3755e8aa46ab99585987ae7d1e1816ac6139"} Oct 31 17:42:51.702951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.702897 7618 scope.go:122] "RemoveContainer" containerID="c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3" Oct 31 17:42:51.704739 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.704714 7618 generic.go:356] "Generic (PLEG): container finished" podID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerID="c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c" exitCode=0 Oct 31 17:42:51.704843 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.704768 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" Oct 31 17:42:51.704843 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.704795 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" event={"ID":"58baa22b-b233-4796-82d2-983a7ec80b0b","Type":"ContainerDied","Data":"c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c"} Oct 31 17:42:51.704843 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.704828 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr" event={"ID":"58baa22b-b233-4796-82d2-983a7ec80b0b","Type":"ContainerDied","Data":"305ee961096a2985a660cabdbb137c72cc59adc28a2b4da0061d347a955982b1"} Oct 31 17:42:51.727731 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.727592 7618 scope.go:122] "RemoveContainer" containerID="c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3" Oct 31 17:42:51.732300 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:51.730601 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3\": container with ID starting with c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3 not found: ID does not exist" containerID="c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3" Oct 31 17:42:51.732300 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.730640 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3"} err="failed to get container status \"c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3\": rpc error: code = NotFound desc = could not find container \"c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3\": container with ID starting with c902f537a5220cfe7722d2720ac29111f313fe7a7e16db5a17cbcd20958d20e3 not found: ID does not exist" Oct 31 17:42:51.732300 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.730664 7618 scope.go:122] "RemoveContainer" containerID="c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c" Oct 31 17:42:51.757369 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.757344 7618 scope.go:122] "RemoveContainer" containerID="c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c" Oct 31 17:42:51.759099 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:51.759068 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c\": container with ID starting with c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c not found: ID does not exist" containerID="c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c" Oct 31 17:42:51.759230 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.759104 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c"} err="failed to get container status \"c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c\": rpc error: code = NotFound desc = could not find container \"c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c\": container with ID starting with c7f08d04c307005e73ea360bd73e9e151305259942cff3522799ac56e874df5c not found: ID does not exist" Oct 31 17:42:51.778202 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.778167 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr"] Oct 31 17:42:51.788779 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.788738 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/distinct-2-w6-d80080-kserve-79788fbcdd-8gldr"] Oct 31 17:42:51.809649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.809595 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q"] Oct 31 17:42:51.813520 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:51.813488 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/simulator-w6-d80080-kserve-c7885fcb5-kmm7q"] Oct 31 17:42:52.302319 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:52.301906 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq"] Oct 31 17:42:52.302806 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:52.302653 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" containerID="cri-o://162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44" gracePeriod=30 Oct 31 17:42:52.309869 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:52.309846 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4"] Oct 31 17:42:52.310396 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:52.310366 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" containerID="cri-o://e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0" gracePeriod=30 Oct 31 17:42:52.352277 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:52.335221 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 17:42:52.352791 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:52.352754 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" podUID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 17:42:52.364511 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:52.364465 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn"] Oct 31 17:42:52.364727 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:52.364697 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" containerName="istio-proxy" containerID="cri-o://4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885" gracePeriod=30 Oct 31 17:42:52.375332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:52.375302 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" path="/var/lib/kubelet/pods/58baa22b-b233-4796-82d2-983a7ec80b0b/volumes" Oct 31 17:42:52.375856 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:52.375836 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" path="/var/lib/kubelet/pods/67be76a3-2fc6-4e3e-8690-7f6549760cea/volumes" Oct 31 17:42:53.322713 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:53.322674 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 17:42:53.346415 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:53.346361 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" podUID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 17:42:54.316318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:54.316279 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 17:42:54.316505 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:54.316342 7618 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" podUID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 17:42:55.322646 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:55.322617 7618 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:55.323202 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:55.323174 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:55.323967 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:55.323945 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:42:55.606110 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:55.606021 7618 patch_prober.go:29] interesting pod/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:42:55.606110 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:55.606102 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:42:57.598904 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.598508 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:42:57.613225 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.613177 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-kube-api-access-bq2l2\" (UniqueName: \"kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-kube-api-access-bq2l2\") pod \"fb83a702-5210-41d1-b099-1c7d515c99d7\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " Oct 31 17:42:57.613403 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.613288 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-podinfo\") pod \"fb83a702-5210-41d1-b099-1c7d515c99d7\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " Oct 31 17:42:57.613403 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.613335 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-token\" (UniqueName: \"kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-token\") pod \"fb83a702-5210-41d1-b099-1c7d515c99d7\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " Oct 31 17:42:57.613403 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.613354 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-socket\") pod \"fb83a702-5210-41d1-b099-1c7d515c99d7\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " Oct 31 17:42:57.613403 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.613371 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-envoy\") pod \"fb83a702-5210-41d1-b099-1c7d515c99d7\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " Oct 31 17:42:57.613689 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.613467 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-data\") pod \"fb83a702-5210-41d1-b099-1c7d515c99d7\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " Oct 31 17:42:57.613689 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.613495 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb83a702-5210-41d1-b099-1c7d515c99d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb83a702-5210-41d1-b099-1c7d515c99d7-istiod-ca-cert\") pod \"fb83a702-5210-41d1-b099-1c7d515c99d7\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " Oct 31 17:42:57.613689 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.613515 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-credential-socket\") pod \"fb83a702-5210-41d1-b099-1c7d515c99d7\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " Oct 31 17:42:57.613689 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.613531 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-certs\") pod \"fb83a702-5210-41d1-b099-1c7d515c99d7\" (UID: \"fb83a702-5210-41d1-b099-1c7d515c99d7\") " Oct 31 17:42:57.614148 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.614127 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-certs" pod "fb83a702-5210-41d1-b099-1c7d515c99d7" (UID: "fb83a702-5210-41d1-b099-1c7d515c99d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.614426 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.614400 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-credential-socket" pod "fb83a702-5210-41d1-b099-1c7d515c99d7" (UID: "fb83a702-5210-41d1-b099-1c7d515c99d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.614526 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.614407 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb83a702-5210-41d1-b099-1c7d515c99d7-istiod-ca-cert" pod "fb83a702-5210-41d1-b099-1c7d515c99d7" (UID: "fb83a702-5210-41d1-b099-1c7d515c99d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:57.615166 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.615132 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-socket" pod "fb83a702-5210-41d1-b099-1c7d515c99d7" (UID: "fb83a702-5210-41d1-b099-1c7d515c99d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.615372 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.615222 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-data" pod "fb83a702-5210-41d1-b099-1c7d515c99d7" (UID: "fb83a702-5210-41d1-b099-1c7d515c99d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.620857 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.620803 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-kube-api-access-bq2l2" pod "fb83a702-5210-41d1-b099-1c7d515c99d7" (UID: "fb83a702-5210-41d1-b099-1c7d515c99d7"). InnerVolumeSpecName "kube-api-access-bq2l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:57.621188 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.621158 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-podinfo" pod "fb83a702-5210-41d1-b099-1c7d515c99d7" (UID: "fb83a702-5210-41d1-b099-1c7d515c99d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:42:57.623150 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.623123 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-token" pod "fb83a702-5210-41d1-b099-1c7d515c99d7" (UID: "fb83a702-5210-41d1-b099-1c7d515c99d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:57.624008 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.623980 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-envoy" pod "fb83a702-5210-41d1-b099-1c7d515c99d7" (UID: "fb83a702-5210-41d1-b099-1c7d515c99d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.628813 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.628234 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:42:57.715922 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.715855 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb83a702-5210-41d1-b099-1c7d515c99d7-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.715922 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.715896 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.715922 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.715912 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.715922 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.715926 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq2l2\" (UniqueName: \"kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-kube-api-access-bq2l2\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.716310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.715941 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.716310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.715955 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.716310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.715969 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.716310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.715984 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.716310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.715997 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb83a702-5210-41d1-b099-1c7d515c99d7-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.747285 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.747154 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:42:57.831446 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.831303 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-socket\") pod \"92160482-537c-4d9d-b55c-e8200f78f40a\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " Oct 31 17:42:57.831951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.831905 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-socket" pod "92160482-537c-4d9d-b55c-e8200f78f40a" (UID: "92160482-537c-4d9d-b55c-e8200f78f40a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.840951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.832094 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-data\") pod \"92160482-537c-4d9d-b55c-e8200f78f40a\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " Oct 31 17:42:57.840951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.832144 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-envoy\") pod \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " Oct 31 17:42:57.840951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.832177 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-token\" (UniqueName: \"kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-token\") pod \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " Oct 31 17:42:57.840951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.832264 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-kube-api-access-j6lpx\" (UniqueName: \"kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-kube-api-access-j6lpx\") pod \"92160482-537c-4d9d-b55c-e8200f78f40a\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " Oct 31 17:42:57.840951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.832308 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-credential-socket\") pod \"92160482-537c-4d9d-b55c-e8200f78f40a\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " Oct 31 17:42:57.840951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.832453 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-istio-token\" (UniqueName: \"kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-istio-token\") pod \"92160482-537c-4d9d-b55c-e8200f78f40a\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " Oct 31 17:42:57.840951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.832483 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-envoy\") pod \"92160482-537c-4d9d-b55c-e8200f78f40a\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " Oct 31 17:42:57.840951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.832524 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-socket\") pod \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " Oct 31 17:42:57.840951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.832579 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-podinfo\") pod \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " Oct 31 17:42:57.840951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.832628 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-credential-socket\") pod \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " Oct 31 17:42:57.840951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.832661 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-data\") pod \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " Oct 31 17:42:57.840951 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.832716 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92160482-537c-4d9d-b55c-e8200f78f40a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92160482-537c-4d9d-b55c-e8200f78f40a-istio-podinfo\") pod \"92160482-537c-4d9d-b55c-e8200f78f40a\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " Oct 31 17:42:57.853971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.846137 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-data" pod "92160482-537c-4d9d-b55c-e8200f78f40a" (UID: "92160482-537c-4d9d-b55c-e8200f78f40a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.853971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.849019 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-token" pod "9e715d3f-f0cf-43ab-8fc0-b6413bb55451" (UID: "9e715d3f-f0cf-43ab-8fc0-b6413bb55451"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:57.853971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.850419 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-envoy" pod "92160482-537c-4d9d-b55c-e8200f78f40a" (UID: "92160482-537c-4d9d-b55c-e8200f78f40a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.853971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.850670 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-data" pod "9e715d3f-f0cf-43ab-8fc0-b6413bb55451" (UID: "9e715d3f-f0cf-43ab-8fc0-b6413bb55451"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.853971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.850912 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-credential-socket" pod "9e715d3f-f0cf-43ab-8fc0-b6413bb55451" (UID: "9e715d3f-f0cf-43ab-8fc0-b6413bb55451"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.853971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.851363 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-envoy" pod "9e715d3f-f0cf-43ab-8fc0-b6413bb55451" (UID: "9e715d3f-f0cf-43ab-8fc0-b6413bb55451"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.853971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.851527 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-podinfo" pod "9e715d3f-f0cf-43ab-8fc0-b6413bb55451" (UID: "9e715d3f-f0cf-43ab-8fc0-b6413bb55451"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:42:57.853971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.851586 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-socket" pod "9e715d3f-f0cf-43ab-8fc0-b6413bb55451" (UID: "9e715d3f-f0cf-43ab-8fc0-b6413bb55451"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.853971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.851803 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-credential-socket" pod "92160482-537c-4d9d-b55c-e8200f78f40a" (UID: "92160482-537c-4d9d-b55c-e8200f78f40a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.853971 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.852084 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-certs" pod "9e715d3f-f0cf-43ab-8fc0-b6413bb55451" (UID: "9e715d3f-f0cf-43ab-8fc0-b6413bb55451"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.854685 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-certs\") pod \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.854808 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92160482-537c-4d9d-b55c-e8200f78f40a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92160482-537c-4d9d-b55c-e8200f78f40a-istiod-ca-cert\") pod \"92160482-537c-4d9d-b55c-e8200f78f40a\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.854874 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-certs\") pod \"92160482-537c-4d9d-b55c-e8200f78f40a\" (UID: \"92160482-537c-4d9d-b55c-e8200f78f40a\") " Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.854909 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-kube-api-access-mx986\" (UniqueName: \"kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-kube-api-access-mx986\") pod \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.854959 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istiod-ca-cert\") pod \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\" (UID: \"9e715d3f-f0cf-43ab-8fc0-b6413bb55451\") " Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855179 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-certs" pod "92160482-537c-4d9d-b55c-e8200f78f40a" (UID: "92160482-537c-4d9d-b55c-e8200f78f40a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855186 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92160482-537c-4d9d-b55c-e8200f78f40a-istiod-ca-cert" pod "92160482-537c-4d9d-b55c-e8200f78f40a" (UID: "92160482-537c-4d9d-b55c-e8200f78f40a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855589 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istiod-ca-cert" pod "9e715d3f-f0cf-43ab-8fc0-b6413bb55451" (UID: "9e715d3f-f0cf-43ab-8fc0-b6413bb55451"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855721 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855771 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855786 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855799 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92160482-537c-4d9d-b55c-e8200f78f40a-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855811 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855823 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855838 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855851 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855863 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855876 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855890 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92160482-537c-4d9d-b55c-e8200f78f40a-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855904 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.855955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.855920 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.857084 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.856082 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-istio-token" pod "92160482-537c-4d9d-b55c-e8200f78f40a" (UID: "92160482-537c-4d9d-b55c-e8200f78f40a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:57.857084 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.856141 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-kube-api-access-j6lpx" pod "92160482-537c-4d9d-b55c-e8200f78f40a" (UID: "92160482-537c-4d9d-b55c-e8200f78f40a"). InnerVolumeSpecName "kube-api-access-j6lpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:57.857084 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.856223 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92160482-537c-4d9d-b55c-e8200f78f40a-istio-podinfo" pod "92160482-537c-4d9d-b55c-e8200f78f40a" (UID: "92160482-537c-4d9d-b55c-e8200f78f40a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:42:57.858546 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.858499 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-kube-api-access-mx986" pod "9e715d3f-f0cf-43ab-8fc0-b6413bb55451" (UID: "9e715d3f-f0cf-43ab-8fc0-b6413bb55451"). InnerVolumeSpecName "kube-api-access-mx986". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:57.871756 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.864323 7618 generic.go:356] "Generic (PLEG): container finished" podID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerID="162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44" exitCode=0 Oct 31 17:42:57.871756 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.868943 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" event={"ID":"9e715d3f-f0cf-43ab-8fc0-b6413bb55451","Type":"ContainerDied","Data":"162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44"} Oct 31 17:42:57.871756 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.869026 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" event={"ID":"9e715d3f-f0cf-43ab-8fc0-b6413bb55451","Type":"ContainerDied","Data":"00c26cce37d0cb13ec8fa4c236a6f34eac145329536dd68b01332e3b1ebdf492"} Oct 31 17:42:57.871756 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.869063 7618 scope.go:122] "RemoveContainer" containerID="162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44" Oct 31 17:42:57.871756 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.869243 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq" Oct 31 17:42:57.875346 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.875129 7618 generic.go:356] "Generic (PLEG): container finished" podID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerID="e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0" exitCode=0 Oct 31 17:42:57.879407 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.875682 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" event={"ID":"fb83a702-5210-41d1-b099-1c7d515c99d7","Type":"ContainerDied","Data":"e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0"} Oct 31 17:42:57.879407 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.875745 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" event={"ID":"fb83a702-5210-41d1-b099-1c7d515c99d7","Type":"ContainerDied","Data":"84024fb9d807180391d8b9d44c2545ebfcdfa8b89dec728d4aef75f337e32594"} Oct 31 17:42:57.879407 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.875837 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4" Oct 31 17:42:57.923317 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.898528 7618 generic.go:356] "Generic (PLEG): container finished" podID="92160482-537c-4d9d-b55c-e8200f78f40a" containerID="4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885" exitCode=0 Oct 31 17:42:57.923317 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.898586 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" event={"ID":"92160482-537c-4d9d-b55c-e8200f78f40a","Type":"ContainerDied","Data":"4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885"} Oct 31 17:42:57.923317 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.898636 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" event={"ID":"92160482-537c-4d9d-b55c-e8200f78f40a","Type":"ContainerDied","Data":"e49c0fb171d3ef615d1a7374ec0195f07b97fff05067d38ccad3d992083578b3"} Oct 31 17:42:57.923317 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.898717 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn" Oct 31 17:42:57.959475 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.954718 7618 scope.go:122] "RemoveContainer" containerID="162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44" Oct 31 17:42:57.959475 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:57.955125 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44\": container with ID starting with 162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44 not found: ID does not exist" containerID="162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44" Oct 31 17:42:57.959475 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.955163 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44"} err="failed to get container status \"162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44\": rpc error: code = NotFound desc = could not find container \"162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44\": container with ID starting with 162f5e1a4c8c0c1feb478299406c79ddf29f0f63794804d1627902f600ddbc44 not found: ID does not exist" Oct 31 17:42:57.959475 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.955187 7618 scope.go:122] "RemoveContainer" containerID="e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0" Oct 31 17:42:57.959475 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.955667 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq"] Oct 31 17:42:57.959475 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.958371 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92160482-537c-4d9d-b55c-e8200f78f40a-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.959475 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.958413 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx986\" (UniqueName: \"kubernetes.io/projected/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-kube-api-access-mx986\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.959475 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.958431 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e715d3f-f0cf-43ab-8fc0-b6413bb55451-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.959475 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.958446 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6lpx\" (UniqueName: \"kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-kube-api-access-j6lpx\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.959475 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.958463 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92160482-537c-4d9d-b55c-e8200f78f40a-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:57.962609 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:57.961160 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-vhtxq"] Oct 31 17:42:58.041562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.035446 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4"] Oct 31 17:42:58.041562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.036660 7618 scope.go:122] "RemoveContainer" containerID="e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0" Oct 31 17:42:58.112285 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:58.097115 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0\": container with ID starting with e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0 not found: ID does not exist" containerID="e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0" Oct 31 17:42:58.112285 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.097167 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0"} err="failed to get container status \"e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0\": rpc error: code = NotFound desc = could not find container \"e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0\": container with ID starting with e0ee018da633a9f24b2645cdae1aabd46f36113ed572230a2200465fc90125e0 not found: ID does not exist" Oct 31 17:42:58.112285 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.097197 7618 scope.go:122] "RemoveContainer" containerID="4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885" Oct 31 17:42:58.155425 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.155386 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-6wnv4"] Oct 31 17:42:58.163358 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.163314 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn"] Oct 31 17:42:58.174927 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.172769 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e59ca-openshift-default-7cd8df4957-8vqzn"] Oct 31 17:42:58.176466 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.176440 7618 scope.go:122] "RemoveContainer" containerID="4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885" Oct 31 17:42:58.177034 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:58.177009 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885\": container with ID starting with 4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885 not found: ID does not exist" containerID="4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885" Oct 31 17:42:58.177224 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.177163 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885"} err="failed to get container status \"4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885\": rpc error: code = NotFound desc = could not find container \"4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885\": container with ID starting with 4a7b9d6988bb739f44fb6765c884909ae42fe9d64f05a7b71cd0a1ec77664885 not found: ID does not exist" Oct 31 17:42:58.399598 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.399446 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" path="/var/lib/kubelet/pods/92160482-537c-4d9d-b55c-e8200f78f40a/volumes" Oct 31 17:42:58.400858 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.400703 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" path="/var/lib/kubelet/pods/9e715d3f-f0cf-43ab-8fc0-b6413bb55451/volumes" Oct 31 17:42:58.402005 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.401965 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" path="/var/lib/kubelet/pods/fb83a702-5210-41d1-b099-1c7d515c99d7/volumes" Oct 31 17:42:58.403099 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.403064 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp"] Oct 31 17:42:58.515886 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:58.514886 7618 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-serving-cert: secret "maas-api-e2e-worker-w0-ee816e-serving-cert" not found Oct 31 17:42:58.516195 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:58.516177 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls podName:00fd9edb-9e10-4cac-8b27-a036c2718fce nodeName:}" failed. No retries permitted until 2026-10-31 17:42:59.016150422 +0000 UTC m=+900.764124881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls") pod "maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" (UID: "00fd9edb-9e10-4cac-8b27-a036c2718fce") : secret "maas-api-e2e-worker-w0-ee816e-serving-cert" not found Oct 31 17:42:58.658280 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.658151 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" containerID="cri-o://efac4f278c3a2021b406ac0396922eb5b265fbd1b172fa687c00e9f34643fb2b" gracePeriod=37 Oct 31 17:42:58.938921 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.933790 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4"] Oct 31 17:42:58.938921 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.936860 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1c0bfd-qsdd4"] Oct 31 17:42:58.975528 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.975491 7618 generic.go:356] "Generic (PLEG): container finished" podID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerID="efac4f278c3a2021b406ac0396922eb5b265fbd1b172fa687c00e9f34643fb2b" exitCode=0 Oct 31 17:42:58.975803 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.975777 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" podUID="00fd9edb-9e10-4cac-8b27-a036c2718fce" containerName="maas-api" containerID="cri-o://ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02" gracePeriod=30 Oct 31 17:42:58.976231 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:58.976200 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" event={"ID":"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5","Type":"ContainerDied","Data":"efac4f278c3a2021b406ac0396922eb5b265fbd1b172fa687c00e9f34643fb2b"} Oct 31 17:42:59.003184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.003148 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:42:59.041435 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.039473 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:42:59.127713 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:59.115123 7618 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-serving-cert: secret "maas-api-e2e-worker-w0-ee816e-serving-cert" not found Oct 31 17:42:59.127713 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:42:59.115202 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls podName:00fd9edb-9e10-4cac-8b27-a036c2718fce nodeName:}" failed. No retries permitted until 2026-10-31 17:43:00.115179137 +0000 UTC m=+901.863153584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls") pod "maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" (UID: "00fd9edb-9e10-4cac-8b27-a036c2718fce") : secret "maas-api-e2e-worker-w0-ee816e-serving-cert" not found Oct 31 17:42:59.611279 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.609578 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:42:59.734278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.713640 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-dshm\") pod \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " Oct 31 17:42:59.734278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.713738 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-kube-api-access-fdhls\" (UniqueName: \"kubernetes.io/projected/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-kube-api-access-fdhls\") pod \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " Oct 31 17:42:59.734278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.713775 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tls-certs\") pod \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " Oct 31 17:42:59.734278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.713912 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-home\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-home\") pod \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " Oct 31 17:42:59.734278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.713946 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tmp-dir\") pod \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " Oct 31 17:42:59.734278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.714123 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-model-cache\") pod \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " Oct 31 17:42:59.734278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.721568 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-home" pod "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" (UID: "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:59.734278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.732381 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tmp-dir" pod "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" (UID: "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:59.734278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.733278 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-model-cache" pod "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" (UID: "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:59.816335 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.815863 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-kube-api-access-fdhls" pod "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" (UID: "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5"). InnerVolumeSpecName "kube-api-access-fdhls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:42:59.833535 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.826701 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-dshm" pod "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" (UID: "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:59.833535 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.828524 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-dshm\") pod \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\" (UID: \"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5\") " Oct 31 17:42:59.833535 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.829096 7618 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-home\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:59.833535 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.829120 7618 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tmp-dir\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:59.833535 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.829134 7618 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-model-cache\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:59.833535 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.829149 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdhls\" (UniqueName: \"kubernetes.io/projected/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-kube-api-access-fdhls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:59.833535 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:42:59.829220 7618 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5/volumes/kubernetes.io~empty-dir/dshm Oct 31 17:42:59.833535 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.829234 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-dshm" pod "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" (UID: "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:42:59.894184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.893999 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tls-certs" pod "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" (UID: "dfbf57d8-cd64-45ae-9cc0-7b65a52207b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:42:59.918648 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.913900 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:42:59.930533 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.930502 7618 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-dshm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:42:59.930753 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:42:59.930736 7618 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5-tls-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:00.007312 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.007273 7618 generic.go:356] "Generic (PLEG): container finished" podID="00fd9edb-9e10-4cac-8b27-a036c2718fce" containerID="ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02" exitCode=0 Oct 31 17:43:00.007785 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.007749 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" event={"ID":"00fd9edb-9e10-4cac-8b27-a036c2718fce","Type":"ContainerDied","Data":"ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02"} Oct 31 17:43:00.007903 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.007868 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" event={"ID":"00fd9edb-9e10-4cac-8b27-a036c2718fce","Type":"ContainerDied","Data":"1005cf2d6e5299a1c681956cbd002004fed5e2fcf885d2cc8c85407f44174a7c"} Oct 31 17:43:00.007987 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.007930 7618 scope.go:122] "RemoveContainer" containerID="ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02" Oct 31 17:43:00.009292 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.008957 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp" Oct 31 17:43:00.012804 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.012781 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" event={"ID":"dfbf57d8-cd64-45ae-9cc0-7b65a52207b5","Type":"ContainerDied","Data":"0d36c104ae65c93e50327495cf1616a4ae7c3573f94d63eca8423797f9c0ebd7"} Oct 31 17:43:00.013158 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.012979 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" Oct 31 17:43:00.027982 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.027638 7618 scope.go:122] "RemoveContainer" containerID="ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02" Oct 31 17:43:00.028108 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:00.028025 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02\": container with ID starting with ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02 not found: ID does not exist" containerID="ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02" Oct 31 17:43:00.028108 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.028055 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02"} err="failed to get container status \"ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02\": rpc error: code = NotFound desc = could not find container \"ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02\": container with ID starting with ac816ff17c821f04ac8ef122c2aa58757efb2c2b8b54594652d687d9e06c5f02 not found: ID does not exist" Oct 31 17:43:00.028108 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.028073 7618 scope.go:122] "RemoveContainer" containerID="efac4f278c3a2021b406ac0396922eb5b265fbd1b172fa687c00e9f34643fb2b" Oct 31 17:43:00.032542 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.031444 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls\") pod \"00fd9edb-9e10-4cac-8b27-a036c2718fce\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " Oct 31 17:43:00.032542 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.031544 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-metrics-certs\") pod \"00fd9edb-9e10-4cac-8b27-a036c2718fce\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " Oct 31 17:43:00.032542 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.031597 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00fd9edb-9e10-4cac-8b27-a036c2718fce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fd9edb-9e10-4cac-8b27-a036c2718fce-openshift-service-ca\") pod \"00fd9edb-9e10-4cac-8b27-a036c2718fce\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " Oct 31 17:43:00.032542 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.031716 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fd9edb-9e10-4cac-8b27-a036c2718fce-kube-api-access-zgkbr\" (UniqueName: \"kubernetes.io/projected/00fd9edb-9e10-4cac-8b27-a036c2718fce-kube-api-access-zgkbr\") pod \"00fd9edb-9e10-4cac-8b27-a036c2718fce\" (UID: \"00fd9edb-9e10-4cac-8b27-a036c2718fce\") " Oct 31 17:43:00.033482 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.033183 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fd9edb-9e10-4cac-8b27-a036c2718fce-openshift-service-ca" pod "00fd9edb-9e10-4cac-8b27-a036c2718fce" (UID: "00fd9edb-9e10-4cac-8b27-a036c2718fce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:00.037403 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.037378 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-metrics-certs" pod "00fd9edb-9e10-4cac-8b27-a036c2718fce" (UID: "00fd9edb-9e10-4cac-8b27-a036c2718fce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:43:00.038845 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.038823 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls" pod "00fd9edb-9e10-4cac-8b27-a036c2718fce" (UID: "00fd9edb-9e10-4cac-8b27-a036c2718fce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:43:00.043686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.043650 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fd9edb-9e10-4cac-8b27-a036c2718fce-kube-api-access-zgkbr" pod "00fd9edb-9e10-4cac-8b27-a036c2718fce" (UID: "00fd9edb-9e10-4cac-8b27-a036c2718fce"). InnerVolumeSpecName "kube-api-access-zgkbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:00.093974 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.093885 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr"] Oct 31 17:43:00.099613 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.099576 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr"] Oct 31 17:43:00.133686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.133647 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgkbr\" (UniqueName: \"kubernetes.io/projected/00fd9edb-9e10-4cac-8b27-a036c2718fce-kube-api-access-zgkbr\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:00.133686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.133680 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:00.133686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.133692 7618 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fd9edb-9e10-4cac-8b27-a036c2718fce-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:00.133960 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.133702 7618 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fd9edb-9e10-4cac-8b27-a036c2718fce-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:00.210356 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.210303 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d80080/premium-w6-d80080-kserve-675dc985dd-l2zdr" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 17:43:00.228647 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.228602 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8"] Oct 31 17:43:00.236066 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.236029 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d80080-w6nl8"] Oct 31 17:43:00.354436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.354341 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp"] Oct 31 17:43:00.357928 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.357894 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ee816e-76f98c8b49-jrnjp"] Oct 31 17:43:00.373355 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.373318 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fd9edb-9e10-4cac-8b27-a036c2718fce" path="/var/lib/kubelet/pods/00fd9edb-9e10-4cac-8b27-a036c2718fce/volumes" Oct 31 17:43:00.373886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.373865 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54de558b-f489-46a4-8978-de66c15583d7" path="/var/lib/kubelet/pods/54de558b-f489-46a4-8978-de66c15583d7/volumes" Oct 31 17:43:00.374371 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.374353 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1" path="/var/lib/kubelet/pods/c156fbd5-d2eb-400a-b8f2-5a3a98aff4e1/volumes" Oct 31 17:43:00.375121 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:00.375101 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" path="/var/lib/kubelet/pods/dfbf57d8-cd64-45ae-9cc0-7b65a52207b5/volumes" Oct 31 17:43:06.337360 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:06.337308 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp"] Oct 31 17:43:06.338828 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:06.338755 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerName="istio-proxy" containerID="cri-o://53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3" gracePeriod=30 Oct 31 17:43:06.379010 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:06.378977 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb"] Oct 31 17:43:06.379160 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:06.379039 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75"] Oct 31 17:43:06.379284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:06.379236 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" podUID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" containerName="istio-proxy" containerID="cri-o://7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf" gracePeriod=30 Oct 31 17:43:06.379410 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:06.379306 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" containerID="cri-o://515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8" gracePeriod=30 Oct 31 17:43:08.337038 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:08.337001 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw"] Oct 31 17:43:08.337483 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:08.337306 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" containerID="cri-o://cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e" gracePeriod=30 Oct 31 17:43:08.471278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:08.466104 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t"] Oct 31 17:43:08.471278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:08.471166 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" podUID="288b5eba-33dc-4576-b70a-0e1b4da73004" containerName="istio-proxy" containerID="cri-o://d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243" gracePeriod=30 Oct 31 17:43:08.472178 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:08.471595 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb"] Oct 31 17:43:08.472178 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:08.471905 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" podUID="1f68a151-a3f2-4609-beab-4cbe07577b59" containerName="istio-proxy" containerID="cri-o://62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f" gracePeriod=30 Oct 31 17:43:08.578346 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:08.578309 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:43:09.160077 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:09.160032 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7"] Oct 31 17:43:09.160382 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:09.160348 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" containerID="cri-o://b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66" gracePeriod=30 Oct 31 17:43:09.205662 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:09.205623 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2"] Oct 31 17:43:09.205960 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:09.205880 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" containerID="cri-o://5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79" gracePeriod=30 Oct 31 17:43:09.210605 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:09.210578 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b"] Oct 31 17:43:09.210823 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:09.210745 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" podUID="446910ca-78af-4c6f-8196-56c209b33636" containerName="istio-proxy" containerID="cri-o://544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0" gracePeriod=30 Oct 31 17:43:09.691670 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:09.691632 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:43:09.692122 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:09.691699 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" podUID="1f68a151-a3f2-4609-beab-4cbe07577b59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:43:09.980973 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:09.980887 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:43:09.980973 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:09.980955 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" podUID="288b5eba-33dc-4576-b70a-0e1b4da73004" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:43:10.129857 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:10.129818 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:43:10.130010 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:10.129887 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" podUID="446910ca-78af-4c6f-8196-56c209b33636" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:43:10.325644 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:10.325559 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:43:10.325644 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:10.325620 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" podUID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:43:10.575367 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:10.575296 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l"] Oct 31 17:43:10.577475 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:10.577384 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" containerID="cri-o://933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362" gracePeriod=30 Oct 31 17:43:10.589382 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:10.589342 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs"] Oct 31 17:43:10.589604 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:10.589579 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" podUID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" containerName="istio-proxy" containerID="cri-o://a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3" gracePeriod=30 Oct 31 17:43:10.601826 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:10.601791 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p"] Oct 31 17:43:10.602003 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:10.601982 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" containerID="cri-o://c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e" gracePeriod=30 Oct 31 17:43:11.557753 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.557726 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:43:11.612552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.612016 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-envoy\") pod \"82a1b216-1644-4e44-ae2c-4e23c06d7940\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " Oct 31 17:43:11.612552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.612079 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-data\") pod \"82a1b216-1644-4e44-ae2c-4e23c06d7940\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " Oct 31 17:43:11.612552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.612119 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-kube-api-access-ljsvs\" (UniqueName: \"kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-kube-api-access-ljsvs\") pod \"82a1b216-1644-4e44-ae2c-4e23c06d7940\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " Oct 31 17:43:11.612552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.612143 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-credential-socket\") pod \"82a1b216-1644-4e44-ae2c-4e23c06d7940\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " Oct 31 17:43:11.612552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.612213 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-podinfo\") pod \"82a1b216-1644-4e44-ae2c-4e23c06d7940\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " Oct 31 17:43:11.612552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.612241 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-certs\") pod \"82a1b216-1644-4e44-ae2c-4e23c06d7940\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " Oct 31 17:43:11.612552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.612288 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-token\" (UniqueName: \"kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-token\") pod \"82a1b216-1644-4e44-ae2c-4e23c06d7940\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " Oct 31 17:43:11.612552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.612331 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-socket\") pod \"82a1b216-1644-4e44-ae2c-4e23c06d7940\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " Oct 31 17:43:11.612552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.612371 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a1b216-1644-4e44-ae2c-4e23c06d7940-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a1b216-1644-4e44-ae2c-4e23c06d7940-istiod-ca-cert\") pod \"82a1b216-1644-4e44-ae2c-4e23c06d7940\" (UID: \"82a1b216-1644-4e44-ae2c-4e23c06d7940\") " Oct 31 17:43:11.613719 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.613375 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-data" pod "82a1b216-1644-4e44-ae2c-4e23c06d7940" (UID: "82a1b216-1644-4e44-ae2c-4e23c06d7940"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.613719 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.613398 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a1b216-1644-4e44-ae2c-4e23c06d7940-istiod-ca-cert" pod "82a1b216-1644-4e44-ae2c-4e23c06d7940" (UID: "82a1b216-1644-4e44-ae2c-4e23c06d7940"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:11.613719 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.613394 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-certs" pod "82a1b216-1644-4e44-ae2c-4e23c06d7940" (UID: "82a1b216-1644-4e44-ae2c-4e23c06d7940"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.613719 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.613602 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-credential-socket" pod "82a1b216-1644-4e44-ae2c-4e23c06d7940" (UID: "82a1b216-1644-4e44-ae2c-4e23c06d7940"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.613719 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.613703 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-socket" pod "82a1b216-1644-4e44-ae2c-4e23c06d7940" (UID: "82a1b216-1644-4e44-ae2c-4e23c06d7940"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.615944 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.615915 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-envoy" pod "82a1b216-1644-4e44-ae2c-4e23c06d7940" (UID: "82a1b216-1644-4e44-ae2c-4e23c06d7940"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.617402 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.617357 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-kube-api-access-ljsvs" pod "82a1b216-1644-4e44-ae2c-4e23c06d7940" (UID: "82a1b216-1644-4e44-ae2c-4e23c06d7940"). InnerVolumeSpecName "kube-api-access-ljsvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:11.643451 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.642333 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-token" pod "82a1b216-1644-4e44-ae2c-4e23c06d7940" (UID: "82a1b216-1644-4e44-ae2c-4e23c06d7940"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:11.646757 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.646719 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-podinfo" pod "82a1b216-1644-4e44-ae2c-4e23c06d7940" (UID: "82a1b216-1644-4e44-ae2c-4e23c06d7940"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:43:11.671944 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.671914 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:43:11.695001 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.694976 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:43:11.713668 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.713631 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-data\") pod \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " Oct 31 17:43:11.713853 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.713674 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-credential-socket\") pod \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " Oct 31 17:43:11.713853 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.713790 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-kube-api-access-dgq2l\" (UniqueName: \"kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-kube-api-access-dgq2l\") pod \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " Oct 31 17:43:11.713853 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.713827 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-credential-socket\") pod \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " Oct 31 17:43:11.714037 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.713861 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-podinfo\") pod \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " Oct 31 17:43:11.714037 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.713889 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-certs\") pod \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " Oct 31 17:43:11.714037 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.713917 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istiod-ca-cert\") pod \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " Oct 31 17:43:11.714037 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.713981 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-token\" (UniqueName: \"kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-token\") pod \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " Oct 31 17:43:11.714037 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714013 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-token\" (UniqueName: \"kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-token\") pod \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " Oct 31 17:43:11.714400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714042 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-podinfo\") pod \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " Oct 31 17:43:11.714400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714069 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-credential-socket" pod "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" (UID: "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.714400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714121 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-kube-api-access-hts67\" (UniqueName: \"kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-kube-api-access-hts67\") pod \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " Oct 31 17:43:11.714400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714177 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-envoy\") pod \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " Oct 31 17:43:11.714400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714209 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istiod-ca-cert\") pod \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " Oct 31 17:43:11.714400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714308 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-data\") pod \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " Oct 31 17:43:11.714400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714342 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-certs\") pod \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " Oct 31 17:43:11.714400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714348 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-certs" pod "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" (UID: "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.714400 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714370 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-socket\") pod \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\" (UID: \"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd\") " Oct 31 17:43:11.714887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714423 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-socket\") pod \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " Oct 31 17:43:11.714887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714450 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-envoy\") pod \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\" (UID: \"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7\") " Oct 31 17:43:11.714887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.714877 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-data" pod "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" (UID: "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.715074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715018 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-credential-socket" pod "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" (UID: "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.715138 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715102 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.715138 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715123 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.715267 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715139 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljsvs\" (UniqueName: \"kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-kube-api-access-ljsvs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.715267 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715155 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.715267 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715171 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.715267 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715184 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.715267 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715200 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.715267 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715212 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.715267 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715225 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a1b216-1644-4e44-ae2c-4e23c06d7940-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.715267 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715238 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a1b216-1644-4e44-ae2c-4e23c06d7940-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.715267 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715270 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a1b216-1644-4e44-ae2c-4e23c06d7940-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.715729 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715284 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.715729 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715509 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-certs" pod "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" (UID: "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.715840 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.715818 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istiod-ca-cert" pod "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" (UID: "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:11.717949 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.716052 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-data" pod "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" (UID: "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.717949 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.716519 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-socket" pod "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" (UID: "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.717949 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.716855 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istiod-ca-cert" pod "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" (UID: "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:11.717949 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.717908 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-socket" pod "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" (UID: "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.722453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.722422 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-podinfo" pod "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" (UID: "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:43:11.724634 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.724543 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-podinfo" pod "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" (UID: "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:43:11.724634 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.724572 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-envoy" pod "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" (UID: "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.724634 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.724597 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-token" pod "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" (UID: "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:11.724634 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.724579 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-kube-api-access-hts67" pod "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" (UID: "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7"). InnerVolumeSpecName "kube-api-access-hts67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:11.725131 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.725109 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-envoy" pod "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" (UID: "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:11.725229 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.725143 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-kube-api-access-dgq2l" pod "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" (UID: "625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd"). InnerVolumeSpecName "kube-api-access-dgq2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:11.726889 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.726863 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-token" pod "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" (UID: "4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:11.742663 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.742628 7618 patch_prober.go:29] interesting pod/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:43:11.742795 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.742710 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:43:11.816605 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816563 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816605 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816602 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816616 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816630 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hts67\" (UniqueName: \"kubernetes.io/projected/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-kube-api-access-hts67\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816646 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816659 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816672 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816684 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816697 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816710 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816723 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816736 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgq2l\" (UniqueName: \"kubernetes.io/projected/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-kube-api-access-dgq2l\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816749 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816762 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:11.816850 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:11.816777 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:12.192681 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.192645 7618 generic.go:356] "Generic (PLEG): container finished" podID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerID="515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8" exitCode=0 Oct 31 17:43:12.192892 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.192698 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" Oct 31 17:43:12.192892 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.192731 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" event={"ID":"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd","Type":"ContainerDied","Data":"515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8"} Oct 31 17:43:12.192892 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.192805 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb" event={"ID":"625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd","Type":"ContainerDied","Data":"4321fb08d870e3d548d7bf83a4c09862e863d7569d37dfca13141dd25ba2f8a7"} Oct 31 17:43:12.192892 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.192834 7618 scope.go:122] "RemoveContainer" containerID="515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8" Oct 31 17:43:12.194536 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.194509 7618 generic.go:356] "Generic (PLEG): container finished" podID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerID="53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3" exitCode=0 Oct 31 17:43:12.194717 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.194543 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" event={"ID":"82a1b216-1644-4e44-ae2c-4e23c06d7940","Type":"ContainerDied","Data":"53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3"} Oct 31 17:43:12.194808 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.194754 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" event={"ID":"82a1b216-1644-4e44-ae2c-4e23c06d7940","Type":"ContainerDied","Data":"49646a4e94464caa27209373770060195e106cebad7f78a4e57a3cf7944afabe"} Oct 31 17:43:12.194808 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.194566 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" Oct 31 17:43:12.196493 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.196468 7618 generic.go:356] "Generic (PLEG): container finished" podID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" containerID="7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf" exitCode=0 Oct 31 17:43:12.196685 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.196516 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" Oct 31 17:43:12.196807 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.196519 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" event={"ID":"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7","Type":"ContainerDied","Data":"7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf"} Oct 31 17:43:12.196807 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.196794 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75" event={"ID":"4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7","Type":"ContainerDied","Data":"56765bd84f280d686232fcbcdc60d0fe32f9cdca97f39d1c48c0cd4cfb018592"} Oct 31 17:43:12.206939 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.206920 7618 scope.go:122] "RemoveContainer" containerID="515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8" Oct 31 17:43:12.207284 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:12.207238 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8\": container with ID starting with 515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8 not found: ID does not exist" containerID="515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8" Oct 31 17:43:12.207381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.207283 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8"} err="failed to get container status \"515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8\": rpc error: code = NotFound desc = could not find container \"515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8\": container with ID starting with 515051c1236c9802a78fb2a40f4eced7a650e4f8ffbf1861f13febcb799d7ea8 not found: ID does not exist" Oct 31 17:43:12.207381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.207306 7618 scope.go:122] "RemoveContainer" containerID="53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3" Oct 31 17:43:12.217934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.217904 7618 scope.go:122] "RemoveContainer" containerID="53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3" Oct 31 17:43:12.218291 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:12.218264 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3\": container with ID starting with 53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3 not found: ID does not exist" containerID="53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3" Oct 31 17:43:12.218381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.218301 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3"} err="failed to get container status \"53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3\": rpc error: code = NotFound desc = could not find container \"53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3\": container with ID starting with 53344917ce483ed6e0dd14a62c8661b742ac4716e6ea48d5da7f4ff986d140f3 not found: ID does not exist" Oct 31 17:43:12.218381 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.218319 7618 scope.go:122] "RemoveContainer" containerID="7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf" Oct 31 17:43:12.228152 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.228064 7618 scope.go:122] "RemoveContainer" containerID="7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf" Oct 31 17:43:12.228607 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:12.228577 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf\": container with ID starting with 7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf not found: ID does not exist" containerID="7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf" Oct 31 17:43:12.228725 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.228624 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf"} err="failed to get container status \"7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf\": rpc error: code = NotFound desc = could not find container \"7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf\": container with ID starting with 7d99abcdcb12335559bf7796c6ddd351c3158c5f0c3dc89cb798c47815f7edbf not found: ID does not exist" Oct 31 17:43:12.255156 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.255118 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75"] Oct 31 17:43:12.264961 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.264929 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-shr75"] Oct 31 17:43:12.296270 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.296209 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb"] Oct 31 17:43:12.309098 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.309058 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-wnpxb"] Oct 31 17:43:12.338740 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.338708 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp"] Oct 31 17:43:12.342022 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.341992 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp"] Oct 31 17:43:12.374934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.374904 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" path="/var/lib/kubelet/pods/4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7/volumes" Oct 31 17:43:12.375424 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.375406 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" path="/var/lib/kubelet/pods/625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd/volumes" Oct 31 17:43:12.375845 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.375830 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" path="/var/lib/kubelet/pods/82a1b216-1644-4e44-ae2c-4e23c06d7940/volumes" Oct 31 17:43:12.488454 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.488363 7618 patch_prober.go:29] interesting pod/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:43:12.488454 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:12.488437 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b78c31-openshift-default-774cb4f6c9-z2dwp" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:43:13.119151 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.119106 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:43:13.119584 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.119216 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:43:13.632008 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.631978 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:43:13.742835 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.742784 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-envoy\") pod \"f70d0930-55ba-4c3d-9b94-046606c04bf0\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " Oct 31 17:43:13.743013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.742857 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-certs\") pod \"f70d0930-55ba-4c3d-9b94-046606c04bf0\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " Oct 31 17:43:13.743013 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.742885 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-token\" (UniqueName: \"kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-token\") pod \"f70d0930-55ba-4c3d-9b94-046606c04bf0\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " Oct 31 17:43:13.743136 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.743052 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-credential-socket\") pod \"f70d0930-55ba-4c3d-9b94-046606c04bf0\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " Oct 31 17:43:13.743136 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.743094 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-data\") pod \"f70d0930-55ba-4c3d-9b94-046606c04bf0\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " Oct 31 17:43:13.743282 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.743151 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-socket\") pod \"f70d0930-55ba-4c3d-9b94-046606c04bf0\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " Oct 31 17:43:13.743348 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.743294 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-kube-api-access-rzxl7\" (UniqueName: \"kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-kube-api-access-rzxl7\") pod \"f70d0930-55ba-4c3d-9b94-046606c04bf0\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " Oct 31 17:43:13.743348 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.743306 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-certs" pod "f70d0930-55ba-4c3d-9b94-046606c04bf0" (UID: "f70d0930-55ba-4c3d-9b94-046606c04bf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.743348 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.743334 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70d0930-55ba-4c3d-9b94-046606c04bf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70d0930-55ba-4c3d-9b94-046606c04bf0-istiod-ca-cert\") pod \"f70d0930-55ba-4c3d-9b94-046606c04bf0\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " Oct 31 17:43:13.743514 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.743369 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-podinfo\") pod \"f70d0930-55ba-4c3d-9b94-046606c04bf0\" (UID: \"f70d0930-55ba-4c3d-9b94-046606c04bf0\") " Oct 31 17:43:13.744799 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.743573 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-data" pod "f70d0930-55ba-4c3d-9b94-046606c04bf0" (UID: "f70d0930-55ba-4c3d-9b94-046606c04bf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.744799 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.744465 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-socket" pod "f70d0930-55ba-4c3d-9b94-046606c04bf0" (UID: "f70d0930-55ba-4c3d-9b94-046606c04bf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.744799 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.744533 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-credential-socket" pod "f70d0930-55ba-4c3d-9b94-046606c04bf0" (UID: "f70d0930-55ba-4c3d-9b94-046606c04bf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.744799 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.744753 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70d0930-55ba-4c3d-9b94-046606c04bf0-istiod-ca-cert" pod "f70d0930-55ba-4c3d-9b94-046606c04bf0" (UID: "f70d0930-55ba-4c3d-9b94-046606c04bf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:13.746585 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.746562 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-token" pod "f70d0930-55ba-4c3d-9b94-046606c04bf0" (UID: "f70d0930-55ba-4c3d-9b94-046606c04bf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:13.746730 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.746711 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.746730 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.746728 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.746930 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.746739 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.746930 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.746751 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70d0930-55ba-4c3d-9b94-046606c04bf0-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.746930 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.746772 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.746930 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.746784 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.747067 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.746999 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-kube-api-access-rzxl7" pod "f70d0930-55ba-4c3d-9b94-046606c04bf0" (UID: "f70d0930-55ba-4c3d-9b94-046606c04bf0"). InnerVolumeSpecName "kube-api-access-rzxl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:13.747165 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.747140 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-podinfo" pod "f70d0930-55ba-4c3d-9b94-046606c04bf0" (UID: "f70d0930-55ba-4c3d-9b94-046606c04bf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:43:13.747512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.747488 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-envoy" pod "f70d0930-55ba-4c3d-9b94-046606c04bf0" (UID: "f70d0930-55ba-4c3d-9b94-046606c04bf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.786159 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.786134 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:43:13.792189 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.792168 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:43:13.848201 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848159 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-token\" (UniqueName: \"kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-token\") pod \"1f68a151-a3f2-4609-beab-4cbe07577b59\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " Oct 31 17:43:13.848432 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848235 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-credential-socket\") pod \"288b5eba-33dc-4576-b70a-0e1b4da73004\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " Oct 31 17:43:13.848432 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848301 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-data\") pod \"1f68a151-a3f2-4609-beab-4cbe07577b59\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " Oct 31 17:43:13.848432 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848337 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-certs\") pod \"1f68a151-a3f2-4609-beab-4cbe07577b59\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " Oct 31 17:43:13.848432 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848388 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-kube-api-access-pvnxf\" (UniqueName: \"kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-kube-api-access-pvnxf\") pod \"1f68a151-a3f2-4609-beab-4cbe07577b59\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " Oct 31 17:43:13.848432 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848422 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-podinfo\") pod \"288b5eba-33dc-4576-b70a-0e1b4da73004\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " Oct 31 17:43:13.848722 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848483 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f68a151-a3f2-4609-beab-4cbe07577b59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f68a151-a3f2-4609-beab-4cbe07577b59-istiod-ca-cert\") pod \"1f68a151-a3f2-4609-beab-4cbe07577b59\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " Oct 31 17:43:13.848722 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848528 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-data\") pod \"288b5eba-33dc-4576-b70a-0e1b4da73004\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " Oct 31 17:43:13.848722 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848581 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-token\" (UniqueName: \"kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-token\") pod \"288b5eba-33dc-4576-b70a-0e1b4da73004\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " Oct 31 17:43:13.848722 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848616 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-socket\") pod \"288b5eba-33dc-4576-b70a-0e1b4da73004\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " Oct 31 17:43:13.848931 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848728 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-certs\") pod \"288b5eba-33dc-4576-b70a-0e1b4da73004\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " Oct 31 17:43:13.848931 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848779 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-data" pod "1f68a151-a3f2-4609-beab-4cbe07577b59" (UID: "1f68a151-a3f2-4609-beab-4cbe07577b59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.848931 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848806 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-kube-api-access-vvqgh\" (UniqueName: \"kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-kube-api-access-vvqgh\") pod \"288b5eba-33dc-4576-b70a-0e1b4da73004\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " Oct 31 17:43:13.848931 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848846 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-data" pod "288b5eba-33dc-4576-b70a-0e1b4da73004" (UID: "288b5eba-33dc-4576-b70a-0e1b4da73004"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.848931 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848874 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f68a151-a3f2-4609-beab-4cbe07577b59-istiod-ca-cert" pod "1f68a151-a3f2-4609-beab-4cbe07577b59" (UID: "1f68a151-a3f2-4609-beab-4cbe07577b59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:13.848931 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848905 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-envoy\") pod \"1f68a151-a3f2-4609-beab-4cbe07577b59\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " Oct 31 17:43:13.849232 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.848968 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-credential-socket\") pod \"1f68a151-a3f2-4609-beab-4cbe07577b59\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " Oct 31 17:43:13.849232 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.849021 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288b5eba-33dc-4576-b70a-0e1b4da73004-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288b5eba-33dc-4576-b70a-0e1b4da73004-istiod-ca-cert\") pod \"288b5eba-33dc-4576-b70a-0e1b4da73004\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " Oct 31 17:43:13.849232 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.849097 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-certs" pod "1f68a151-a3f2-4609-beab-4cbe07577b59" (UID: "1f68a151-a3f2-4609-beab-4cbe07577b59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.849232 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.849120 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-socket" pod "288b5eba-33dc-4576-b70a-0e1b4da73004" (UID: "288b5eba-33dc-4576-b70a-0e1b4da73004"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.849232 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.849183 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-certs" pod "288b5eba-33dc-4576-b70a-0e1b4da73004" (UID: "288b5eba-33dc-4576-b70a-0e1b4da73004"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.849515 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.849389 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-credential-socket" pod "1f68a151-a3f2-4609-beab-4cbe07577b59" (UID: "1f68a151-a3f2-4609-beab-4cbe07577b59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.849515 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.849394 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-envoy\") pod \"288b5eba-33dc-4576-b70a-0e1b4da73004\" (UID: \"288b5eba-33dc-4576-b70a-0e1b4da73004\") " Oct 31 17:43:13.849515 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.849440 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-podinfo\") pod \"1f68a151-a3f2-4609-beab-4cbe07577b59\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " Oct 31 17:43:13.849515 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.849483 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-socket\") pod \"1f68a151-a3f2-4609-beab-4cbe07577b59\" (UID: \"1f68a151-a3f2-4609-beab-4cbe07577b59\") " Oct 31 17:43:13.849717 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.849675 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-credential-socket" pod "288b5eba-33dc-4576-b70a-0e1b4da73004" (UID: "288b5eba-33dc-4576-b70a-0e1b4da73004"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.850056 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850029 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzxl7\" (UniqueName: \"kubernetes.io/projected/f70d0930-55ba-4c3d-9b94-046606c04bf0-kube-api-access-rzxl7\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.850156 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850060 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.850156 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850076 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.850156 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850089 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70d0930-55ba-4c3d-9b94-046606c04bf0-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.850156 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850128 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.850156 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850140 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.850156 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850152 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f68a151-a3f2-4609-beab-4cbe07577b59-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.850521 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850163 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.850521 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850192 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.850521 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850206 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.850521 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850221 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.850521 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850247 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288b5eba-33dc-4576-b70a-0e1b4da73004-istiod-ca-cert" pod "288b5eba-33dc-4576-b70a-0e1b4da73004" (UID: "288b5eba-33dc-4576-b70a-0e1b4da73004"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:13.850521 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.850432 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-socket" pod "1f68a151-a3f2-4609-beab-4cbe07577b59" (UID: "1f68a151-a3f2-4609-beab-4cbe07577b59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.851405 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.851364 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-kube-api-access-pvnxf" pod "1f68a151-a3f2-4609-beab-4cbe07577b59" (UID: "1f68a151-a3f2-4609-beab-4cbe07577b59"). InnerVolumeSpecName "kube-api-access-pvnxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:13.851511 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.851417 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-kube-api-access-vvqgh" pod "288b5eba-33dc-4576-b70a-0e1b4da73004" (UID: "288b5eba-33dc-4576-b70a-0e1b4da73004"). InnerVolumeSpecName "kube-api-access-vvqgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:13.852741 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.852709 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-envoy" pod "1f68a151-a3f2-4609-beab-4cbe07577b59" (UID: "1f68a151-a3f2-4609-beab-4cbe07577b59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.852741 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.852718 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-token" pod "1f68a151-a3f2-4609-beab-4cbe07577b59" (UID: "1f68a151-a3f2-4609-beab-4cbe07577b59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:13.852917 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.852774 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-podinfo" pod "1f68a151-a3f2-4609-beab-4cbe07577b59" (UID: "1f68a151-a3f2-4609-beab-4cbe07577b59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:43:13.852917 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.852855 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-podinfo" pod "288b5eba-33dc-4576-b70a-0e1b4da73004" (UID: "288b5eba-33dc-4576-b70a-0e1b4da73004"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:43:13.853522 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.853502 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-token" pod "288b5eba-33dc-4576-b70a-0e1b4da73004" (UID: "288b5eba-33dc-4576-b70a-0e1b4da73004"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:13.853993 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.853969 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-envoy" pod "288b5eba-33dc-4576-b70a-0e1b4da73004" (UID: "288b5eba-33dc-4576-b70a-0e1b4da73004"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:13.951748 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.951707 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvqgh\" (UniqueName: \"kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-kube-api-access-vvqgh\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.951748 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.951742 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.951748 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.951754 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288b5eba-33dc-4576-b70a-0e1b4da73004-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.952000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.951762 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.952000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.951773 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.952000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.951781 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f68a151-a3f2-4609-beab-4cbe07577b59-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.952000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.951790 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.952000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.951803 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvnxf\" (UniqueName: \"kubernetes.io/projected/1f68a151-a3f2-4609-beab-4cbe07577b59-kube-api-access-pvnxf\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.952000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.951814 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.952000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.951825 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288b5eba-33dc-4576-b70a-0e1b4da73004-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:13.983407 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.983319 7618 patch_prober.go:29] interesting pod/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:43:13.983407 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:13.983384 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:43:14.217346 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.217236 7618 generic.go:356] "Generic (PLEG): container finished" podID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerID="cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e" exitCode=0 Oct 31 17:43:14.217346 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.217328 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" event={"ID":"f70d0930-55ba-4c3d-9b94-046606c04bf0","Type":"ContainerDied","Data":"cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e"} Oct 31 17:43:14.217346 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.217426 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" event={"ID":"f70d0930-55ba-4c3d-9b94-046606c04bf0","Type":"ContainerDied","Data":"b2ee0607e919008efa6be6a81f63a830f53a79c6cb8f274b3fbdd3ae0771fcc9"} Oct 31 17:43:14.217346 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.217345 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw" Oct 31 17:43:14.218117 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.217491 7618 scope.go:122] "RemoveContainer" containerID="cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e" Oct 31 17:43:14.219798 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.219775 7618 generic.go:356] "Generic (PLEG): container finished" podID="288b5eba-33dc-4576-b70a-0e1b4da73004" containerID="d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243" exitCode=0 Oct 31 17:43:14.219987 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.219889 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" Oct 31 17:43:14.219987 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.219949 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" event={"ID":"288b5eba-33dc-4576-b70a-0e1b4da73004","Type":"ContainerDied","Data":"d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243"} Oct 31 17:43:14.220109 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.220079 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t" event={"ID":"288b5eba-33dc-4576-b70a-0e1b4da73004","Type":"ContainerDied","Data":"57d6b3032ae35388a4b7dadb4a2c989e8efcff9a472c54eba75029de884b9a66"} Oct 31 17:43:14.223161 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.223131 7618 generic.go:356] "Generic (PLEG): container finished" podID="1f68a151-a3f2-4609-beab-4cbe07577b59" containerID="62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f" exitCode=0 Oct 31 17:43:14.223365 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.223235 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" event={"ID":"1f68a151-a3f2-4609-beab-4cbe07577b59","Type":"ContainerDied","Data":"62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f"} Oct 31 17:43:14.223365 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.223290 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" event={"ID":"1f68a151-a3f2-4609-beab-4cbe07577b59","Type":"ContainerDied","Data":"a98b21fb8260c5320bd91832436e74926cd66fbbf0835ba9dac37e41ee0bde21"} Oct 31 17:43:14.223811 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.223789 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb" Oct 31 17:43:14.299035 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.299001 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb"] Oct 31 17:43:14.303695 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.303663 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-x69bb"] Oct 31 17:43:14.331217 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.330925 7618 scope.go:122] "RemoveContainer" containerID="cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e" Oct 31 17:43:14.331371 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:14.331342 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e\": container with ID starting with cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e not found: ID does not exist" containerID="cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e" Oct 31 17:43:14.331459 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.331385 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e"} err="failed to get container status \"cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e\": rpc error: code = NotFound desc = could not find container \"cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e\": container with ID starting with cf93a079da833f808128870d51b1310b7f13846f64424b5fe1862e50ce22237e not found: ID does not exist" Oct 31 17:43:14.331459 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.331415 7618 scope.go:122] "RemoveContainer" containerID="d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243" Oct 31 17:43:14.335106 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.335044 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw"] Oct 31 17:43:14.342607 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.340401 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-swwtw"] Oct 31 17:43:14.347431 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.347141 7618 scope.go:122] "RemoveContainer" containerID="d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243" Oct 31 17:43:14.347842 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:14.347708 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243\": container with ID starting with d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243 not found: ID does not exist" containerID="d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243" Oct 31 17:43:14.347842 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.347748 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243"} err="failed to get container status \"d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243\": rpc error: code = NotFound desc = could not find container \"d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243\": container with ID starting with d29dd2258374afbdae142b90edb6817eb4dd60c850dd6ca9b337ba81492c7243 not found: ID does not exist" Oct 31 17:43:14.347842 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.347771 7618 scope.go:122] "RemoveContainer" containerID="62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f" Oct 31 17:43:14.381360 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.381329 7618 scope.go:122] "RemoveContainer" containerID="62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f" Oct 31 17:43:14.381664 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.381637 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f68a151-a3f2-4609-beab-4cbe07577b59" path="/var/lib/kubelet/pods/1f68a151-a3f2-4609-beab-4cbe07577b59/volumes" Oct 31 17:43:14.382203 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.382182 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" path="/var/lib/kubelet/pods/f70d0930-55ba-4c3d-9b94-046606c04bf0/volumes" Oct 31 17:43:14.382542 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.382524 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t"] Oct 31 17:43:14.383158 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:14.383136 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f\": container with ID starting with 62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f not found: ID does not exist" containerID="62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f" Oct 31 17:43:14.383247 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.383168 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f"} err="failed to get container status \"62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f\": rpc error: code = NotFound desc = could not find container \"62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f\": container with ID starting with 62136aa9327f1f5e07db6d45bb108b06d4987aae71b9ac85f544b35a88e2521f not found: ID does not exist" Oct 31 17:43:14.385064 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.385045 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1c0bfd-openshift-default-f7f657c57-6vh4t"] Oct 31 17:43:14.488884 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.488822 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:43:14.558632 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.558599 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-kube-api-access-q2xjj\" (UniqueName: \"kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-kube-api-access-q2xjj\") pod \"446910ca-78af-4c6f-8196-56c209b33636\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " Oct 31 17:43:14.558817 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.558646 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/446910ca-78af-4c6f-8196-56c209b33636-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446910ca-78af-4c6f-8196-56c209b33636-istio-podinfo\") pod \"446910ca-78af-4c6f-8196-56c209b33636\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " Oct 31 17:43:14.558817 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.558691 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-certs\") pod \"446910ca-78af-4c6f-8196-56c209b33636\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " Oct 31 17:43:14.558817 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.558719 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/446910ca-78af-4c6f-8196-56c209b33636-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446910ca-78af-4c6f-8196-56c209b33636-istiod-ca-cert\") pod \"446910ca-78af-4c6f-8196-56c209b33636\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " Oct 31 17:43:14.558817 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.558772 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-socket\") pod \"446910ca-78af-4c6f-8196-56c209b33636\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " Oct 31 17:43:14.559049 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.558924 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-data\") pod \"446910ca-78af-4c6f-8196-56c209b33636\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " Oct 31 17:43:14.559049 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.558970 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-istio-token\" (UniqueName: \"kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-istio-token\") pod \"446910ca-78af-4c6f-8196-56c209b33636\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " Oct 31 17:43:14.559049 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.559007 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-credential-socket\") pod \"446910ca-78af-4c6f-8196-56c209b33636\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " Oct 31 17:43:14.559209 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.559083 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-envoy\") pod \"446910ca-78af-4c6f-8196-56c209b33636\" (UID: \"446910ca-78af-4c6f-8196-56c209b33636\") " Oct 31 17:43:14.559209 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.559086 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-socket" pod "446910ca-78af-4c6f-8196-56c209b33636" (UID: "446910ca-78af-4c6f-8196-56c209b33636"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.559209 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.559180 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-data" pod "446910ca-78af-4c6f-8196-56c209b33636" (UID: "446910ca-78af-4c6f-8196-56c209b33636"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.559418 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.559230 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446910ca-78af-4c6f-8196-56c209b33636-istiod-ca-cert" pod "446910ca-78af-4c6f-8196-56c209b33636" (UID: "446910ca-78af-4c6f-8196-56c209b33636"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:14.559418 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.559266 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-certs" pod "446910ca-78af-4c6f-8196-56c209b33636" (UID: "446910ca-78af-4c6f-8196-56c209b33636"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.559418 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.559368 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-credential-socket" pod "446910ca-78af-4c6f-8196-56c209b33636" (UID: "446910ca-78af-4c6f-8196-56c209b33636"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.559714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.559692 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.559714 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.559716 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446910ca-78af-4c6f-8196-56c209b33636-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.560306 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.559728 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.560306 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.559744 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.560306 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.559756 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.562326 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.561973 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/446910ca-78af-4c6f-8196-56c209b33636-istio-podinfo" pod "446910ca-78af-4c6f-8196-56c209b33636" (UID: "446910ca-78af-4c6f-8196-56c209b33636"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:43:14.562326 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.562307 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-istio-token" pod "446910ca-78af-4c6f-8196-56c209b33636" (UID: "446910ca-78af-4c6f-8196-56c209b33636"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:14.562433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.562351 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-kube-api-access-q2xjj" pod "446910ca-78af-4c6f-8196-56c209b33636" (UID: "446910ca-78af-4c6f-8196-56c209b33636"). InnerVolumeSpecName "kube-api-access-q2xjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:14.562545 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.562527 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-envoy" pod "446910ca-78af-4c6f-8196-56c209b33636" (UID: "446910ca-78af-4c6f-8196-56c209b33636"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.565282 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.565241 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:43:14.580092 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.580070 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:43:14.660706 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.660669 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b39ed1-7e86-42d4-9241-94a7e369638b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b39ed1-7e86-42d4-9241-94a7e369638b-istiod-ca-cert\") pod \"61b39ed1-7e86-42d4-9241-94a7e369638b\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " Oct 31 17:43:14.660896 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.660727 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-socket\") pod \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " Oct 31 17:43:14.660896 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.660758 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-data\") pod \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " Oct 31 17:43:14.660896 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.660821 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-credential-socket\") pod \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " Oct 31 17:43:14.660896 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.660851 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istiod-ca-cert\") pod \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " Oct 31 17:43:14.660896 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.660885 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-podinfo\") pod \"61b39ed1-7e86-42d4-9241-94a7e369638b\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " Oct 31 17:43:14.661153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.660945 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-certs\") pod \"61b39ed1-7e86-42d4-9241-94a7e369638b\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " Oct 31 17:43:14.661153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.660982 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-certs\") pod \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " Oct 31 17:43:14.661153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.661042 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-kube-api-access-wjn6m\" (UniqueName: \"kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-kube-api-access-wjn6m\") pod \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " Oct 31 17:43:14.661153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.661105 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-envoy\") pod \"61b39ed1-7e86-42d4-9241-94a7e369638b\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " Oct 31 17:43:14.661153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.661137 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-socket\") pod \"61b39ed1-7e86-42d4-9241-94a7e369638b\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " Oct 31 17:43:14.661450 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.661174 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b39ed1-7e86-42d4-9241-94a7e369638b-istiod-ca-cert" pod "61b39ed1-7e86-42d4-9241-94a7e369638b" (UID: "61b39ed1-7e86-42d4-9241-94a7e369638b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:14.661450 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.661235 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-token\" (UniqueName: \"kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-token\") pod \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " Oct 31 17:43:14.661450 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.661289 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-data\") pod \"61b39ed1-7e86-42d4-9241-94a7e369638b\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " Oct 31 17:43:14.661450 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.661327 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-credential-socket\") pod \"61b39ed1-7e86-42d4-9241-94a7e369638b\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " Oct 31 17:43:14.661450 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.661357 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-envoy\") pod \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " Oct 31 17:43:14.661450 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.661389 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-podinfo\") pod \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\" (UID: \"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2\") " Oct 31 17:43:14.661450 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.661418 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-kube-api-access-jtd7w\" (UniqueName: \"kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-kube-api-access-jtd7w\") pod \"61b39ed1-7e86-42d4-9241-94a7e369638b\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " Oct 31 17:43:14.661450 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.661438 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-token\" (UniqueName: \"kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-token\") pod \"61b39ed1-7e86-42d4-9241-94a7e369638b\" (UID: \"61b39ed1-7e86-42d4-9241-94a7e369638b\") " Oct 31 17:43:14.661925 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.661901 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.662083 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.662061 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446910ca-78af-4c6f-8196-56c209b33636-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.662173 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.662086 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2xjj\" (UniqueName: \"kubernetes.io/projected/446910ca-78af-4c6f-8196-56c209b33636-kube-api-access-q2xjj\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.662173 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.662092 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-socket" pod "61b39ed1-7e86-42d4-9241-94a7e369638b" (UID: "61b39ed1-7e86-42d4-9241-94a7e369638b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.662173 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.662102 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446910ca-78af-4c6f-8196-56c209b33636-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.662173 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.662130 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-certs" pod "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" (UID: "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.662173 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.662141 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b39ed1-7e86-42d4-9241-94a7e369638b-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.662550 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.662311 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-certs" pod "61b39ed1-7e86-42d4-9241-94a7e369638b" (UID: "61b39ed1-7e86-42d4-9241-94a7e369638b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.662550 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.662407 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-credential-socket" pod "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" (UID: "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.662550 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.662442 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-data" pod "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" (UID: "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.662710 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.662555 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-socket" pod "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" (UID: "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.662776 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.662712 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istiod-ca-cert" pod "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" (UID: "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:14.663007 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.662982 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-data" pod "61b39ed1-7e86-42d4-9241-94a7e369638b" (UID: "61b39ed1-7e86-42d4-9241-94a7e369638b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.663191 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.663170 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-credential-socket" pod "61b39ed1-7e86-42d4-9241-94a7e369638b" (UID: "61b39ed1-7e86-42d4-9241-94a7e369638b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.665411 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.665390 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-podinfo" pod "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" (UID: "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:43:14.666021 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.665997 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-token" pod "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" (UID: "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:14.666946 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.666922 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-token" pod "61b39ed1-7e86-42d4-9241-94a7e369638b" (UID: "61b39ed1-7e86-42d4-9241-94a7e369638b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:14.667046 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.666945 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-kube-api-access-jtd7w" pod "61b39ed1-7e86-42d4-9241-94a7e369638b" (UID: "61b39ed1-7e86-42d4-9241-94a7e369638b"). InnerVolumeSpecName "kube-api-access-jtd7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:14.667153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.667133 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-envoy" pod "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" (UID: "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.667595 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.667561 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-envoy" pod "61b39ed1-7e86-42d4-9241-94a7e369638b" (UID: "61b39ed1-7e86-42d4-9241-94a7e369638b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:14.667595 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.667570 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-podinfo" pod "61b39ed1-7e86-42d4-9241-94a7e369638b" (UID: "61b39ed1-7e86-42d4-9241-94a7e369638b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:43:14.668733 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.668705 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-kube-api-access-wjn6m" pod "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" (UID: "c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2"). InnerVolumeSpecName "kube-api-access-wjn6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:14.763125 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763036 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763125 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763081 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763125 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763096 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763125 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763111 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763125 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763125 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763138 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763151 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763165 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjn6m\" (UniqueName: \"kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-kube-api-access-wjn6m\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763178 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763191 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763203 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763216 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763227 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b39ed1-7e86-42d4-9241-94a7e369638b-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763240 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763283 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763300 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtd7w\" (UniqueName: \"kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-kube-api-access-jtd7w\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:14.763538 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:14.763313 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b39ed1-7e86-42d4-9241-94a7e369638b-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:15.233698 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.233653 7618 generic.go:356] "Generic (PLEG): container finished" podID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerID="b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66" exitCode=0 Oct 31 17:43:15.234161 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.233733 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" Oct 31 17:43:15.234161 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.233741 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" event={"ID":"61b39ed1-7e86-42d4-9241-94a7e369638b","Type":"ContainerDied","Data":"b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66"} Oct 31 17:43:15.234161 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.233820 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7" event={"ID":"61b39ed1-7e86-42d4-9241-94a7e369638b","Type":"ContainerDied","Data":"ba35dedaccb5de2c92188be42d0833fe30601c220b372de8c226f6e24eb6ecb3"} Oct 31 17:43:15.234161 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.233877 7618 scope.go:122] "RemoveContainer" containerID="b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66" Oct 31 17:43:15.235676 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.235651 7618 generic.go:356] "Generic (PLEG): container finished" podID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerID="5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79" exitCode=0 Oct 31 17:43:15.235801 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.235711 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" Oct 31 17:43:15.235801 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.235739 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" event={"ID":"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2","Type":"ContainerDied","Data":"5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79"} Oct 31 17:43:15.235801 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.235782 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2" event={"ID":"c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2","Type":"ContainerDied","Data":"10e4fd50ae5f2770a9edfb7dfd793933d3891355b89b31ef4e97d2468b251768"} Oct 31 17:43:15.238193 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.238169 7618 generic.go:356] "Generic (PLEG): container finished" podID="446910ca-78af-4c6f-8196-56c209b33636" containerID="544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0" exitCode=0 Oct 31 17:43:15.238305 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.238203 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" event={"ID":"446910ca-78af-4c6f-8196-56c209b33636","Type":"ContainerDied","Data":"544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0"} Oct 31 17:43:15.238305 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.238244 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" Oct 31 17:43:15.238403 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.238266 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b" event={"ID":"446910ca-78af-4c6f-8196-56c209b33636","Type":"ContainerDied","Data":"75b4e829d1b339ff01c739f8de9e09086984906a062f92e6f7e2f86a3ab18c4d"} Oct 31 17:43:15.244078 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.244034 7618 patch_prober.go:29] interesting pod/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 17:43:15.244195 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.244093 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 17:43:15.247561 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.247544 7618 scope.go:122] "RemoveContainer" containerID="b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66" Oct 31 17:43:15.247871 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:15.247846 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66\": container with ID starting with b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66 not found: ID does not exist" containerID="b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66" Oct 31 17:43:15.247956 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.247883 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66"} err="failed to get container status \"b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66\": rpc error: code = NotFound desc = could not find container \"b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66\": container with ID starting with b7b95d36e451aac9203907d7ef177da835d5ac16796e7f597f35ef52b1c06e66 not found: ID does not exist" Oct 31 17:43:15.247956 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.247908 7618 scope.go:122] "RemoveContainer" containerID="5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79" Oct 31 17:43:15.258072 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.258049 7618 scope.go:122] "RemoveContainer" containerID="5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79" Oct 31 17:43:15.258370 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:15.258350 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79\": container with ID starting with 5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79 not found: ID does not exist" containerID="5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79" Oct 31 17:43:15.258414 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.258377 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79"} err="failed to get container status \"5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79\": rpc error: code = NotFound desc = could not find container \"5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79\": container with ID starting with 5962eeb33dab2e261000b8d694edd363389cc61d0ec615d834dcd13fbaab8f79 not found: ID does not exist" Oct 31 17:43:15.258414 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.258395 7618 scope.go:122] "RemoveContainer" containerID="544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0" Oct 31 17:43:15.267462 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.267444 7618 scope.go:122] "RemoveContainer" containerID="544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0" Oct 31 17:43:15.267707 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:15.267689 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0\": container with ID starting with 544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0 not found: ID does not exist" containerID="544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0" Oct 31 17:43:15.267785 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.267715 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0"} err="failed to get container status \"544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0\": rpc error: code = NotFound desc = could not find container \"544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0\": container with ID starting with 544cdf7595f3ffe0aca0168fd0ea2cfdc18a8a0a08fecdaa3d788e58f3a4a0f0 not found: ID does not exist" Oct 31 17:43:15.311710 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.311667 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2"] Oct 31 17:43:15.320478 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.320452 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-w6vm2"] Oct 31 17:43:15.345410 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.345378 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b"] Oct 31 17:43:15.348605 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.348575 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-fvk9b"] Oct 31 17:43:15.391562 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.391528 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7"] Oct 31 17:43:15.398266 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.398222 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee816e-openshift-default-c4db85b5c-qcgz7"] Oct 31 17:43:15.449190 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.449157 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58b696d4f-lt9f8"] Oct 31 17:43:15.449405 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.449384 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-58b696d4f-lt9f8" podUID="026f1505-1522-4ee3-b908-5737ab6b6526" containerName="manager" containerID="cri-o://dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6" gracePeriod=30 Oct 31 17:43:15.795821 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.795462 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:43:15.874183 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.874153 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-metrics-certs\" (UniqueName: \"kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-metrics-certs\") pod \"026f1505-1522-4ee3-b908-5737ab6b6526\" (UID: \"026f1505-1522-4ee3-b908-5737ab6b6526\") " Oct 31 17:43:15.874344 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.874224 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-webhook-cert\" (UniqueName: \"kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-webhook-cert\") pod \"026f1505-1522-4ee3-b908-5737ab6b6526\" (UID: \"026f1505-1522-4ee3-b908-5737ab6b6526\") " Oct 31 17:43:15.875004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.874442 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026f1505-1522-4ee3-b908-5737ab6b6526-kube-api-access-c475l\" (UniqueName: \"kubernetes.io/projected/026f1505-1522-4ee3-b908-5737ab6b6526-kube-api-access-c475l\") pod \"026f1505-1522-4ee3-b908-5737ab6b6526\" (UID: \"026f1505-1522-4ee3-b908-5737ab6b6526\") " Oct 31 17:43:15.882831 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.882780 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-webhook-cert" pod "026f1505-1522-4ee3-b908-5737ab6b6526" (UID: "026f1505-1522-4ee3-b908-5737ab6b6526"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:43:15.884489 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.884447 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:43:15.886730 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.886688 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-metrics-certs" pod "026f1505-1522-4ee3-b908-5737ab6b6526" (UID: "026f1505-1522-4ee3-b908-5737ab6b6526"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:43:15.886854 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.886757 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026f1505-1522-4ee3-b908-5737ab6b6526-kube-api-access-c475l" pod "026f1505-1522-4ee3-b908-5737ab6b6526" (UID: "026f1505-1522-4ee3-b908-5737ab6b6526"). InnerVolumeSpecName "kube-api-access-c475l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:15.922401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.922357 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:43:15.938797 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.938721 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:43:15.975807 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.975753 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-kube-api-access-bdg4r\" (UniqueName: \"kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-kube-api-access-bdg4r\") pod \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " Oct 31 17:43:15.975807 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.975804 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-token\" (UniqueName: \"kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-token\") pod \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " Oct 31 17:43:15.976047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.975834 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-socket\") pod \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " Oct 31 17:43:15.976047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.975881 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-kube-api-access-d56tp\" (UniqueName: \"kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-kube-api-access-d56tp\") pod \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " Oct 31 17:43:15.976047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.975927 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-podinfo\") pod \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " Oct 31 17:43:15.976047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.975954 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-certs\") pod \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " Oct 31 17:43:15.976047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.975984 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-socket\") pod \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " Oct 31 17:43:15.976047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976024 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-credential-socket\") pod \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " Oct 31 17:43:15.976309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976061 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-credential-socket\") pod \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " Oct 31 17:43:15.976309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976116 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-token\" (UniqueName: \"kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-token\") pod \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " Oct 31 17:43:15.976309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976136 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-data\") pod \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " Oct 31 17:43:15.976309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976154 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-envoy\") pod \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " Oct 31 17:43:15.976309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976181 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istiod-ca-cert\") pod \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " Oct 31 17:43:15.976309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976238 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-certs\") pod \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " Oct 31 17:43:15.976309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976293 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istiod-ca-cert\") pod \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " Oct 31 17:43:15.976539 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976320 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-data\") pod \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\" (UID: \"9ea2ccc1-eee0-4921-9146-78fcd7790bdf\") " Oct 31 17:43:15.976539 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976353 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-podinfo\") pod \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " Oct 31 17:43:15.976539 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976369 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-envoy\") pod \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\" (UID: \"cafa0366-2f1f-4814-b8be-43a4d5540e4c\") " Oct 31 17:43:15.976686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976653 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c475l\" (UniqueName: \"kubernetes.io/projected/026f1505-1522-4ee3-b908-5737ab6b6526-kube-api-access-c475l\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:15.976686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976664 7618 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:15.976686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.976672 7618 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/026f1505-1522-4ee3-b908-5737ab6b6526-webhook-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:15.978049 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.977203 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-credential-socket" pod "cafa0366-2f1f-4814-b8be-43a4d5540e4c" (UID: "cafa0366-2f1f-4814-b8be-43a4d5540e4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:15.978049 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.977366 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istiod-ca-cert" pod "9ea2ccc1-eee0-4921-9146-78fcd7790bdf" (UID: "9ea2ccc1-eee0-4921-9146-78fcd7790bdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:15.978049 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.977585 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-data" pod "9ea2ccc1-eee0-4921-9146-78fcd7790bdf" (UID: "9ea2ccc1-eee0-4921-9146-78fcd7790bdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:15.978049 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.977809 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-data" pod "cafa0366-2f1f-4814-b8be-43a4d5540e4c" (UID: "cafa0366-2f1f-4814-b8be-43a4d5540e4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:15.978049 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.977841 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-certs" pod "cafa0366-2f1f-4814-b8be-43a4d5540e4c" (UID: "cafa0366-2f1f-4814-b8be-43a4d5540e4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:15.978049 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.977849 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istiod-ca-cert" pod "cafa0366-2f1f-4814-b8be-43a4d5540e4c" (UID: "cafa0366-2f1f-4814-b8be-43a4d5540e4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:15.978049 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.978030 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-socket" pod "9ea2ccc1-eee0-4921-9146-78fcd7790bdf" (UID: "9ea2ccc1-eee0-4921-9146-78fcd7790bdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:15.978049 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.978038 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-socket" pod "cafa0366-2f1f-4814-b8be-43a4d5540e4c" (UID: "cafa0366-2f1f-4814-b8be-43a4d5540e4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:15.978576 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.978347 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-credential-socket" pod "9ea2ccc1-eee0-4921-9146-78fcd7790bdf" (UID: "9ea2ccc1-eee0-4921-9146-78fcd7790bdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:15.978912 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.978874 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-certs" pod "9ea2ccc1-eee0-4921-9146-78fcd7790bdf" (UID: "9ea2ccc1-eee0-4921-9146-78fcd7790bdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:15.980009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.979967 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-envoy" pod "cafa0366-2f1f-4814-b8be-43a4d5540e4c" (UID: "cafa0366-2f1f-4814-b8be-43a4d5540e4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:15.980920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.980893 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-token" pod "cafa0366-2f1f-4814-b8be-43a4d5540e4c" (UID: "cafa0366-2f1f-4814-b8be-43a4d5540e4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:15.981006 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.980924 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-podinfo" pod "cafa0366-2f1f-4814-b8be-43a4d5540e4c" (UID: "cafa0366-2f1f-4814-b8be-43a4d5540e4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:43:15.981243 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.981223 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-envoy" pod "9ea2ccc1-eee0-4921-9146-78fcd7790bdf" (UID: "9ea2ccc1-eee0-4921-9146-78fcd7790bdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:15.981243 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.981227 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-kube-api-access-d56tp" pod "9ea2ccc1-eee0-4921-9146-78fcd7790bdf" (UID: "9ea2ccc1-eee0-4921-9146-78fcd7790bdf"). InnerVolumeSpecName "kube-api-access-d56tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:15.981417 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.981395 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-token" pod "9ea2ccc1-eee0-4921-9146-78fcd7790bdf" (UID: "9ea2ccc1-eee0-4921-9146-78fcd7790bdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:15.981727 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.981702 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-podinfo" pod "9ea2ccc1-eee0-4921-9146-78fcd7790bdf" (UID: "9ea2ccc1-eee0-4921-9146-78fcd7790bdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:43:15.983318 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:15.983292 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-kube-api-access-bdg4r" pod "cafa0366-2f1f-4814-b8be-43a4d5540e4c" (UID: "cafa0366-2f1f-4814-b8be-43a4d5540e4c"). InnerVolumeSpecName "kube-api-access-bdg4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:16.077921 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.077820 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-token\" (UniqueName: \"kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-token\") pod \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " Oct 31 17:43:16.077921 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.077870 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-kube-api-access-pmgvb\" (UniqueName: \"kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-kube-api-access-pmgvb\") pod \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " Oct 31 17:43:16.077921 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.077896 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-envoy\") pod \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " Oct 31 17:43:16.078218 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.077923 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-socket\") pod \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " Oct 31 17:43:16.078218 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.077975 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istiod-ca-cert\") pod \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " Oct 31 17:43:16.078218 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078056 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-credential-socket\") pod \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " Oct 31 17:43:16.078218 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078128 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-podinfo\") pod \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " Oct 31 17:43:16.078218 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078164 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-certs\") pod \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " Oct 31 17:43:16.078533 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078287 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-data\") pod \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\" (UID: \"8ccce903-d0aa-4873-adcc-ab8d4d1df86b\") " Oct 31 17:43:16.078887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078685 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.078887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078710 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.078887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078725 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.078887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078740 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.078887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078752 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.078887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078765 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.078887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078776 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.078887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078789 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.078887 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078837 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-socket" pod "8ccce903-d0aa-4873-adcc-ab8d4d1df86b" (UID: "8ccce903-d0aa-4873-adcc-ab8d4d1df86b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079028 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-credential-socket" pod "8ccce903-d0aa-4873-adcc-ab8d4d1df86b" (UID: "8ccce903-d0aa-4873-adcc-ab8d4d1df86b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079105 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istiod-ca-cert" pod "8ccce903-d0aa-4873-adcc-ab8d4d1df86b" (UID: "8ccce903-d0aa-4873-adcc-ab8d4d1df86b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079198 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-data" pod "8ccce903-d0aa-4873-adcc-ab8d4d1df86b" (UID: "8ccce903-d0aa-4873-adcc-ab8d4d1df86b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.078978 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-certs" pod "8ccce903-d0aa-4873-adcc-ab8d4d1df86b" (UID: "8ccce903-d0aa-4873-adcc-ab8d4d1df86b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079233 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079271 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdg4r\" (UniqueName: \"kubernetes.io/projected/cafa0366-2f1f-4814-b8be-43a4d5540e4c-kube-api-access-bdg4r\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079286 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079302 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079317 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d56tp\" (UniqueName: \"kubernetes.io/projected/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-kube-api-access-d56tp\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079333 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079347 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079360 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079374 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea2ccc1-eee0-4921-9146-78fcd7790bdf-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.079457 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.079389 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cafa0366-2f1f-4814-b8be-43a4d5540e4c-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.080536 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.080512 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-token" pod "8ccce903-d0aa-4873-adcc-ab8d4d1df86b" (UID: "8ccce903-d0aa-4873-adcc-ab8d4d1df86b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:16.080944 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.080928 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-podinfo" pod "8ccce903-d0aa-4873-adcc-ab8d4d1df86b" (UID: "8ccce903-d0aa-4873-adcc-ab8d4d1df86b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 17:43:16.081288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.081237 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-envoy" pod "8ccce903-d0aa-4873-adcc-ab8d4d1df86b" (UID: "8ccce903-d0aa-4873-adcc-ab8d4d1df86b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:43:16.081436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.081418 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-kube-api-access-pmgvb" pod "8ccce903-d0aa-4873-adcc-ab8d4d1df86b" (UID: "8ccce903-d0aa-4873-adcc-ab8d4d1df86b"). InnerVolumeSpecName "kube-api-access-pmgvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:43:16.179935 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.179904 7618 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.179935 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.179932 7618 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.179935 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.179941 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmgvb\" (UniqueName: \"kubernetes.io/projected/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-kube-api-access-pmgvb\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.180177 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.179952 7618 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.180177 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.179962 7618 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.180177 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.179970 7618 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.180177 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.179978 7618 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.180177 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.179986 7618 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.180177 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.179993 7618 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ccce903-d0aa-4873-adcc-ab8d4d1df86b-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:43:16.253047 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.253003 7618 generic.go:356] "Generic (PLEG): container finished" podID="026f1505-1522-4ee3-b908-5737ab6b6526" containerID="dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6" exitCode=0 Oct 31 17:43:16.253506 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.253140 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b696d4f-lt9f8" event={"ID":"026f1505-1522-4ee3-b908-5737ab6b6526","Type":"ContainerDied","Data":"dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6"} Oct 31 17:43:16.253506 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.253190 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b696d4f-lt9f8" event={"ID":"026f1505-1522-4ee3-b908-5737ab6b6526","Type":"ContainerDied","Data":"c42309d7fc5c7798470c3927f934fbb67d982bfd0dcf2e2f681a3f600903494e"} Oct 31 17:43:16.253506 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.253219 7618 scope.go:122] "RemoveContainer" containerID="dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6" Oct 31 17:43:16.253506 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.253364 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58b696d4f-lt9f8" Oct 31 17:43:16.255012 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.254899 7618 generic.go:356] "Generic (PLEG): container finished" podID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" containerID="a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3" exitCode=0 Oct 31 17:43:16.255012 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.254951 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" Oct 31 17:43:16.255012 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.254977 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" event={"ID":"8ccce903-d0aa-4873-adcc-ab8d4d1df86b","Type":"ContainerDied","Data":"a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3"} Oct 31 17:43:16.255185 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.255017 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs" event={"ID":"8ccce903-d0aa-4873-adcc-ab8d4d1df86b","Type":"ContainerDied","Data":"5d09289ac3e6d2b107298679125b55e973cfade4e908d774eda0aaf8f1fdff0c"} Oct 31 17:43:16.262649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.262516 7618 generic.go:356] "Generic (PLEG): container finished" podID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerID="933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362" exitCode=0 Oct 31 17:43:16.262649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.262582 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" Oct 31 17:43:16.262649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.262589 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" event={"ID":"cafa0366-2f1f-4814-b8be-43a4d5540e4c","Type":"ContainerDied","Data":"933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362"} Oct 31 17:43:16.262649 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.262639 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l" event={"ID":"cafa0366-2f1f-4814-b8be-43a4d5540e4c","Type":"ContainerDied","Data":"7a561b08167f69196f0509b831453179b01ff7cd8d963504c9f4129ae958f2eb"} Oct 31 17:43:16.265401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.264790 7618 generic.go:356] "Generic (PLEG): container finished" podID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerID="c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e" exitCode=0 Oct 31 17:43:16.265401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.264883 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" event={"ID":"9ea2ccc1-eee0-4921-9146-78fcd7790bdf","Type":"ContainerDied","Data":"c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e"} Oct 31 17:43:16.265401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.264944 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" event={"ID":"9ea2ccc1-eee0-4921-9146-78fcd7790bdf","Type":"ContainerDied","Data":"63bb8eda7b0037b32b8b15823bf7f3644ce28e727f5e558f446180183534322b"} Oct 31 17:43:16.265401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.264980 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p" Oct 31 17:43:16.268477 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.268392 7618 scope.go:122] "RemoveContainer" containerID="8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73" Oct 31 17:43:16.279539 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.279515 7618 scope.go:122] "RemoveContainer" containerID="dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6" Oct 31 17:43:16.279837 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:16.279816 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6\": container with ID starting with dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6 not found: ID does not exist" containerID="dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6" Oct 31 17:43:16.279886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.279846 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6"} err="failed to get container status \"dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6\": rpc error: code = NotFound desc = could not find container \"dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6\": container with ID starting with dc0ee78eb586def2e830e578eae92163d05937c0fce6764b3f86c15d68c218d6 not found: ID does not exist" Oct 31 17:43:16.279886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.279863 7618 scope.go:122] "RemoveContainer" containerID="8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73" Oct 31 17:43:16.280118 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:16.280097 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73\": container with ID starting with 8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73 not found: ID does not exist" containerID="8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73" Oct 31 17:43:16.280178 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.280129 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73"} err="failed to get container status \"8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73\": rpc error: code = NotFound desc = could not find container \"8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73\": container with ID starting with 8237428fa9142fe6dbc55e408732a53d3e00059cd544c71017460b4482b1cb73 not found: ID does not exist" Oct 31 17:43:16.280178 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.280151 7618 scope.go:122] "RemoveContainer" containerID="a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3" Oct 31 17:43:16.289816 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.289794 7618 scope.go:122] "RemoveContainer" containerID="a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3" Oct 31 17:43:16.290147 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:16.290123 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3\": container with ID starting with a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3 not found: ID does not exist" containerID="a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3" Oct 31 17:43:16.290228 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.290154 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3"} err="failed to get container status \"a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3\": rpc error: code = NotFound desc = could not find container \"a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3\": container with ID starting with a9878920a60704e50b77be959fb94942e72f15eac211d7160fa9cf1f223092d3 not found: ID does not exist" Oct 31 17:43:16.290228 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.290173 7618 scope.go:122] "RemoveContainer" containerID="933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362" Oct 31 17:43:16.299961 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.299941 7618 scope.go:122] "RemoveContainer" containerID="933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362" Oct 31 17:43:16.300285 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:16.300239 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362\": container with ID starting with 933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362 not found: ID does not exist" containerID="933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362" Oct 31 17:43:16.300357 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.300294 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362"} err="failed to get container status \"933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362\": rpc error: code = NotFound desc = could not find container \"933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362\": container with ID starting with 933cee5c1814a44ef4217ca45797eaf9bc3c55b60f7702ab0262a80e570f2362 not found: ID does not exist" Oct 31 17:43:16.300357 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.300312 7618 scope.go:122] "RemoveContainer" containerID="c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e" Oct 31 17:43:16.309616 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.309596 7618 scope.go:122] "RemoveContainer" containerID="c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e" Oct 31 17:43:16.309885 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:16.309863 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e\": container with ID starting with c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e not found: ID does not exist" containerID="c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e" Oct 31 17:43:16.309964 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.309895 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e"} err="failed to get container status \"c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e\": rpc error: code = NotFound desc = could not find container \"c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e\": container with ID starting with c3389d015f862a52cccffda44cff71196b036bb46a18f40eab8749f8f5069f5e not found: ID does not exist" Oct 31 17:43:16.347324 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.347229 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p"] Oct 31 17:43:16.358763 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.358727 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-lsr9p"] Oct 31 17:43:16.374122 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.374079 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288b5eba-33dc-4576-b70a-0e1b4da73004" path="/var/lib/kubelet/pods/288b5eba-33dc-4576-b70a-0e1b4da73004/volumes" Oct 31 17:43:16.374858 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.374833 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="446910ca-78af-4c6f-8196-56c209b33636" path="/var/lib/kubelet/pods/446910ca-78af-4c6f-8196-56c209b33636/volumes" Oct 31 17:43:16.375653 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.375632 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" path="/var/lib/kubelet/pods/61b39ed1-7e86-42d4-9241-94a7e369638b/volumes" Oct 31 17:43:16.376770 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.376749 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" path="/var/lib/kubelet/pods/9ea2ccc1-eee0-4921-9146-78fcd7790bdf/volumes" Oct 31 17:43:16.377484 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.377466 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" path="/var/lib/kubelet/pods/c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2/volumes" Oct 31 17:43:16.389290 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.389266 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58b696d4f-lt9f8"] Oct 31 17:43:16.393242 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.393220 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58b696d4f-lt9f8"] Oct 31 17:43:16.433354 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.433317 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l"] Oct 31 17:43:16.439718 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.439691 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-ftn5l"] Oct 31 17:43:16.471310 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.471275 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs"] Oct 31 17:43:16.474163 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:16.474133 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d80080-openshift-default-956cddf57-6kgqs"] Oct 31 17:43:18.372767 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:18.372733 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026f1505-1522-4ee3-b908-5737ab6b6526" path="/var/lib/kubelet/pods/026f1505-1522-4ee3-b908-5737ab6b6526/volumes" Oct 31 17:43:18.373194 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:18.373180 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" path="/var/lib/kubelet/pods/8ccce903-d0aa-4873-adcc-ab8d4d1df86b/volumes" Oct 31 17:43:18.373611 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:18.373599 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" path="/var/lib/kubelet/pods/cafa0366-2f1f-4814-b8be-43a4d5540e4c/volumes" Oct 31 17:43:27.577852 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:27.577817 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:43:38.578144 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:43:38.578106 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:43:38.751308 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:38.751240 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:43:40.031447 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:40.031406 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl"] Oct 31 17:43:40.041573 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:40.041537 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3250a0f-qr9hl"] Oct 31 17:43:40.376071 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:40.375976 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5aed79-4d7b-4c30-b958-afac9cbe69d6" path="/var/lib/kubelet/pods/4c5aed79-4d7b-4c30-b958-afac9cbe69d6/volumes" Oct 31 17:43:42.029550 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:42.029514 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl"] Oct 31 17:43:42.039855 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:42.039819 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8c58c0-8v8sl"] Oct 31 17:43:42.373739 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:42.373647 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a289ed-e730-4e17-a65c-0d2fc1b5593c" path="/var/lib/kubelet/pods/53a289ed-e730-4e17-a65c-0d2fc1b5593c/volumes" Oct 31 17:43:54.025551 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:54.025507 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh"] Oct 31 17:43:54.035956 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:54.035919 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab459dfd-bvdgh"] Oct 31 17:43:54.372668 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:54.372589 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04d20be-e50b-432d-918e-db1854292fc2" path="/var/lib/kubelet/pods/b04d20be-e50b-432d-918e-db1854292fc2/volumes" Oct 31 17:43:56.027353 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:56.027319 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9"] Oct 31 17:43:56.037624 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:56.037595 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970bdea7-hcqp9"] Oct 31 17:43:56.372242 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:56.372149 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e6fc95a-6be4-45da-9ec7-3ae723278112" path="/var/lib/kubelet/pods/7e6fc95a-6be4-45da-9ec7-3ae723278112/volumes" Oct 31 17:43:59.943962 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:43:59.943925 7618 scope.go:122] "RemoveContainer" containerID="d5f5747c0474a8e49f264dd8bcddee1ad1a6b7c813c000536681d92469559eda" Oct 31 17:44:00.023034 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:00.023006 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd"] Oct 31 17:44:00.034486 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:00.034456 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10ac4fb-mpghd"] Oct 31 17:44:00.373441 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:00.373363 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c532ee89-8206-40d2-ab8a-94a259e85911" path="/var/lib/kubelet/pods/c532ee89-8206-40d2-ab8a-94a259e85911/volumes" Oct 31 17:44:02.024717 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:02.024676 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn"] Oct 31 17:44:02.034893 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:02.034860 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b648f55-qtccn"] Oct 31 17:44:02.374317 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:02.374226 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0cb58d2-8678-4a3c-860a-ccedb7f72f18" path="/var/lib/kubelet/pods/d0cb58d2-8678-4a3c-860a-ccedb7f72f18/volumes" Oct 31 17:44:06.025270 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:06.025215 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl"] Oct 31 17:44:06.035088 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:06.035054 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615c753c-fh6zl"] Oct 31 17:44:06.372470 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:06.372392 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5850b4-8cf8-47e6-baeb-2d40def410b9" path="/var/lib/kubelet/pods/3d5850b4-8cf8-47e6-baeb-2d40def410b9/volumes" Oct 31 17:44:08.571501 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:44:08.571463 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:44:13.021208 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:13.021171 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7"] Oct 31 17:44:13.031139 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:13.031112 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef789c02-4tcd7"] Oct 31 17:44:14.024667 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:14.024633 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd"] Oct 31 17:44:14.034904 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:14.034876 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d87b62-wr2jd"] Oct 31 17:44:14.372197 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:14.372118 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09698b60-c579-4a2e-a3ef-124b69b21438" path="/var/lib/kubelet/pods/09698b60-c579-4a2e-a3ef-124b69b21438/volumes" Oct 31 17:44:14.372580 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:14.372565 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a8c99be-1b4f-4031-8126-2a9a1abe3757" path="/var/lib/kubelet/pods/4a8c99be-1b4f-4031-8126-2a9a1abe3757/volumes" Oct 31 17:44:18.024398 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:18.024369 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt"] Oct 31 17:44:18.034686 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:18.034652 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4ade94-l4hqt"] Oct 31 17:44:18.373751 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:18.373663 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1188609b-eccc-4717-8d36-e8252469621e" path="/var/lib/kubelet/pods/1188609b-eccc-4717-8d36-e8252469621e/volumes" Oct 31 17:44:25.024106 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:25.024069 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7"] Oct 31 17:44:25.034186 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:25.034159 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05dc4675-jlms7"] Oct 31 17:44:26.372360 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:26.372327 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51abcbf-e4ec-487f-8c4d-d239cc25a0a2" path="/var/lib/kubelet/pods/f51abcbf-e4ec-487f-8c4d-d239cc25a0a2/volumes" Oct 31 17:44:29.081433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:29.081398 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:44:33.023915 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:33.023880 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h"] Oct 31 17:44:33.034153 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:33.034121 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b72c1ebb-xxs9h"] Oct 31 17:44:34.371997 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:34.371962 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de7eef7-c55f-4da6-be7f-9f41daa2efd3" path="/var/lib/kubelet/pods/4de7eef7-c55f-4da6-be7f-9f41daa2efd3/volumes" Oct 31 17:44:38.572128 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:44:38.572087 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:44:46.444438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:44:46.444400 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:45:00.043447 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:00.043415 7618 scope.go:122] "RemoveContainer" containerID="ce7bb5190a151d9a0583c390122789220e622e51b9a98c19c58ba1904b874d1d" Oct 31 17:45:00.052709 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:00.052686 7618 scope.go:122] "RemoveContainer" containerID="4b1ab4a0f3755252406caa6c229f7df0ab747a4adbbb7823c9c566c01ab2bab1" Oct 31 17:45:00.061140 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:00.061121 7618 scope.go:122] "RemoveContainer" containerID="6d6a30430f8c22030c7131d11d74e4ed79f604110dd58cbf294c9dd503ea7580" Oct 31 17:45:00.069635 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:00.069609 7618 scope.go:122] "RemoveContainer" containerID="8922d6b8a7e89ef825f36624de7d40fa5b7e5969ca1ef8ba49837cc2d233dc23" Oct 31 17:45:00.077640 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:00.077609 7618 scope.go:122] "RemoveContainer" containerID="de025b3b1f3c583cc5e1c5ce896a64e38705c2eeeda3eb158646bcab3c9366cd" Oct 31 17:45:00.085638 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:00.085612 7618 scope.go:122] "RemoveContainer" containerID="b727cfcf7a1960be58d1f6cfb473091cc7f874dd20b098a8c91117d309af8161" Oct 31 17:45:06.323473 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.323435 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn"] Oct 31 17:45:06.324011 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.323991 7618 cpu_manager.go:414] "containerMap: removing container" podUID="1f68a151-a3f2-4609-beab-4cbe07577b59" containerName="istio-proxy" Oct 31 17:45:06.324058 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324015 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f68a151-a3f2-4609-beab-4cbe07577b59" containerName="istio-proxy" Oct 31 17:45:06.324058 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324035 7618 cpu_manager.go:414] "containerMap: removing container" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" Oct 31 17:45:06.324058 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324043 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" Oct 31 17:45:06.324058 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324056 7618 cpu_manager.go:414] "containerMap: removing container" podUID="00fd9edb-9e10-4cac-8b27-a036c2718fce" containerName="maas-api" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324064 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fd9edb-9e10-4cac-8b27-a036c2718fce" containerName="maas-api" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324076 7618 cpu_manager.go:414] "containerMap: removing container" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" containerName="istio-proxy" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324083 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" containerName="istio-proxy" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324093 7618 cpu_manager.go:414] "containerMap: removing container" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324099 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324118 7618 cpu_manager.go:414] "containerMap: removing container" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324126 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324136 7618 cpu_manager.go:414] "containerMap: removing container" podUID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" containerName="istio-proxy" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324143 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" containerName="istio-proxy" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324161 7618 cpu_manager.go:414] "containerMap: removing container" podUID="9d3288d2-a16e-4770-9e3e-b06584d07dd8" containerName="revoke-keys" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324168 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3288d2-a16e-4770-9e3e-b06584d07dd8" containerName="revoke-keys" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324178 7618 cpu_manager.go:414] "containerMap: removing container" podUID="446910ca-78af-4c6f-8196-56c209b33636" containerName="istio-proxy" Oct 31 17:45:06.324184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324184 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446910ca-78af-4c6f-8196-56c209b33636" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324208 7618 cpu_manager.go:414] "containerMap: removing container" podUID="026f1505-1522-4ee3-b908-5737ab6b6526" containerName="manager" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324216 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026f1505-1522-4ee3-b908-5737ab6b6526" containerName="manager" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324226 7618 cpu_manager.go:414] "containerMap: removing container" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324235 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324265 7618 cpu_manager.go:414] "containerMap: removing container" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324272 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324287 7618 cpu_manager.go:414] "containerMap: removing container" podUID="dbf57bc3-4558-446e-96bb-05d057e94b3c" containerName="revoke-keys" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324294 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf57bc3-4558-446e-96bb-05d057e94b3c" containerName="revoke-keys" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324307 7618 cpu_manager.go:414] "containerMap: removing container" podUID="026f1505-1522-4ee3-b908-5737ab6b6526" containerName="manager" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324314 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026f1505-1522-4ee3-b908-5737ab6b6526" containerName="manager" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324328 7618 cpu_manager.go:414] "containerMap: removing container" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324335 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324357 7618 cpu_manager.go:414] "containerMap: removing container" podUID="288b5eba-33dc-4576-b70a-0e1b4da73004" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324365 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288b5eba-33dc-4576-b70a-0e1b4da73004" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324378 7618 cpu_manager.go:414] "containerMap: removing container" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324385 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324398 7618 cpu_manager.go:414] "containerMap: removing container" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324403 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324413 7618 cpu_manager.go:414] "containerMap: removing container" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324417 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324429 7618 cpu_manager.go:414] "containerMap: removing container" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324433 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324439 7618 cpu_manager.go:414] "containerMap: removing container" podUID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324444 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324450 7618 cpu_manager.go:414] "containerMap: removing container" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324454 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324460 7618 cpu_manager.go:414] "containerMap: removing container" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324465 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324471 7618 cpu_manager.go:414] "containerMap: removing container" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324475 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324577 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="288b5eba-33dc-4576-b70a-0e1b4da73004" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324587 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac3a54c-5d8f-4451-9584-dec64d6d235c" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324594 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24f7f5f-1160-4eb1-ad8b-0afb38dd7ea2" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324600 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="58baa22b-b233-4796-82d2-983a7ec80b0b" containerName="main" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324605 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e715d3f-f0cf-43ab-8fc0-b6413bb55451" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324611 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ccce903-d0aa-4873-adcc-ab8d4d1df86b" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324615 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbf57d8-cd64-45ae-9cc0-7b65a52207b5" containerName="main" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324620 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="625c35a6-ff1d-4d4a-aa3c-cd126f7e6dcd" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324627 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a1b216-1644-4e44-ae2c-4e23c06d7940" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324633 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f68a151-a3f2-4609-beab-4cbe07577b59" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324637 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b39ed1-7e86-42d4-9241-94a7e369638b" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324642 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fd9edb-9e10-4cac-8b27-a036c2718fce" containerName="maas-api" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324646 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="92160482-537c-4d9d-b55c-e8200f78f40a" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324651 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea2ccc1-eee0-4921-9146-78fcd7790bdf" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324657 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafa0366-2f1f-4814-b8be-43a4d5540e4c" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324661 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5121d5-3c97-497b-9cb4-8f77aa2cb7b7" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324667 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3288d2-a16e-4770-9e3e-b06584d07dd8" containerName="revoke-keys" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324673 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="446910ca-78af-4c6f-8196-56c209b33636" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324679 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="026f1505-1522-4ee3-b908-5737ab6b6526" containerName="manager" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324684 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb83a702-5210-41d1-b099-1c7d515c99d7" containerName="istio-proxy" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324689 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="67be76a3-2fc6-4e3e-8690-7f6549760cea" containerName="main" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324695 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf57bc3-4558-446e-96bb-05d057e94b3c" containerName="revoke-keys" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324700 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="026f1505-1522-4ee3-b908-5737ab6b6526" containerName="manager" Oct 31 17:45:06.324711 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.324706 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70d0930-55ba-4c3d-9b94-046606c04bf0" containerName="istio-proxy" Oct 31 17:45:06.328072 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.328056 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:06.346422 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.346391 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn"] Oct 31 17:45:06.353355 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.353318 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:45:06.353520 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.353379 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 17:45:06.353520 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.353395 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 17:45:06.353520 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.353441 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:45:06.490600 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.490550 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-ldcpn\" (UID: \"8dc4369c-2baf-495d-84c1-b444f527bfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:06.490789 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.490620 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-ldcpn\" (UID: \"8dc4369c-2baf-495d-84c1-b444f527bfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:06.490789 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.490759 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n54mw\" (UniqueName: \"kubernetes.io/projected/8dc4369c-2baf-495d-84c1-b444f527bfbc-kube-api-access-n54mw\") pod \"maas-api-revoke-sub-ec843018-ldcpn\" (UID: \"8dc4369c-2baf-495d-84c1-b444f527bfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:06.591968 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.591863 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-ldcpn\" (UID: \"8dc4369c-2baf-495d-84c1-b444f527bfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:06.591968 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.591937 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-ldcpn\" (UID: \"8dc4369c-2baf-495d-84c1-b444f527bfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:06.592202 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.592018 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54mw\" (UniqueName: \"kubernetes.io/projected/8dc4369c-2baf-495d-84c1-b444f527bfbc-kube-api-access-n54mw\") pod \"maas-api-revoke-sub-ec843018-ldcpn\" (UID: \"8dc4369c-2baf-495d-84c1-b444f527bfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:06.592635 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.592614 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-ldcpn\" (UID: \"8dc4369c-2baf-495d-84c1-b444f527bfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:06.594580 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.594546 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-ldcpn\" (UID: \"8dc4369c-2baf-495d-84c1-b444f527bfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:06.609539 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.609509 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54mw\" (UniqueName: \"kubernetes.io/projected/8dc4369c-2baf-495d-84c1-b444f527bfbc-kube-api-access-n54mw\") pod \"maas-api-revoke-sub-ec843018-ldcpn\" (UID: \"8dc4369c-2baf-495d-84c1-b444f527bfbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:06.642374 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.642342 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:06.938731 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:06.938691 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn"] Oct 31 17:45:06.943120 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:45:06.943090 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc4369c_2baf_495d_84c1_b444f527bfbc.slice/crio-536b4574493f8bedb29d37cfd465077c1883088865ddaf31f46afa73362ad175 WatchSource:0}: Error finding container 536b4574493f8bedb29d37cfd465077c1883088865ddaf31f46afa73362ad175: Status 404 returned error can't find the container with id 536b4574493f8bedb29d37cfd465077c1883088865ddaf31f46afa73362ad175 Oct 31 17:45:07.084862 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:07.084822 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" event={"ID":"8dc4369c-2baf-495d-84c1-b444f527bfbc","Type":"ContainerStarted","Data":"536b4574493f8bedb29d37cfd465077c1883088865ddaf31f46afa73362ad175"} Oct 31 17:45:08.092424 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:08.092327 7618 generic.go:356] "Generic (PLEG): container finished" podID="8dc4369c-2baf-495d-84c1-b444f527bfbc" containerID="ee78a0df0e3471fa71f3139ffe71091edc5428fe23c659525f148c8e6f91c901" exitCode=0 Oct 31 17:45:08.092802 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:08.092417 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" event={"ID":"8dc4369c-2baf-495d-84c1-b444f527bfbc","Type":"ContainerDied","Data":"ee78a0df0e3471fa71f3139ffe71091edc5428fe23c659525f148c8e6f91c901"} Oct 31 17:45:08.572680 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:45:08.572639 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:45:09.262161 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:09.262136 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:09.325299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:09.324939 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc4369c-2baf-495d-84c1-b444f527bfbc-kube-api-access-n54mw\" (UniqueName: \"kubernetes.io/projected/8dc4369c-2baf-495d-84c1-b444f527bfbc-kube-api-access-n54mw\") pod \"8dc4369c-2baf-495d-84c1-b444f527bfbc\" (UID: \"8dc4369c-2baf-495d-84c1-b444f527bfbc\") " Oct 31 17:45:09.325299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:09.325000 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-openshift-ca\") pod \"8dc4369c-2baf-495d-84c1-b444f527bfbc\" (UID: \"8dc4369c-2baf-495d-84c1-b444f527bfbc\") " Oct 31 17:45:09.325299 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:09.325143 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-xks-ca\") pod \"8dc4369c-2baf-495d-84c1-b444f527bfbc\" (UID: \"8dc4369c-2baf-495d-84c1-b444f527bfbc\") " Oct 31 17:45:09.326441 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:09.326378 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-openshift-ca" pod "8dc4369c-2baf-495d-84c1-b444f527bfbc" (UID: "8dc4369c-2baf-495d-84c1-b444f527bfbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:45:09.328747 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:09.328710 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc4369c-2baf-495d-84c1-b444f527bfbc-kube-api-access-n54mw" pod "8dc4369c-2baf-495d-84c1-b444f527bfbc" (UID: "8dc4369c-2baf-495d-84c1-b444f527bfbc"). InnerVolumeSpecName "kube-api-access-n54mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:45:09.329331 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:09.329302 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-xks-ca" pod "8dc4369c-2baf-495d-84c1-b444f527bfbc" (UID: "8dc4369c-2baf-495d-84c1-b444f527bfbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:45:09.428366 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:09.428317 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n54mw\" (UniqueName: \"kubernetes.io/projected/8dc4369c-2baf-495d-84c1-b444f527bfbc-kube-api-access-n54mw\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:45:09.428366 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:09.428365 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:45:09.428659 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:09.428382 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc4369c-2baf-495d-84c1-b444f527bfbc-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:45:10.108331 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:10.108294 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" event={"ID":"8dc4369c-2baf-495d-84c1-b444f527bfbc","Type":"ContainerDied","Data":"536b4574493f8bedb29d37cfd465077c1883088865ddaf31f46afa73362ad175"} Oct 31 17:45:10.108331 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:10.108320 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn" Oct 31 17:45:10.108555 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:10.108354 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="536b4574493f8bedb29d37cfd465077c1883088865ddaf31f46afa73362ad175" Oct 31 17:45:19.455241 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:19.455203 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:45:25.074223 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:25.074181 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:45:33.024579 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:33.024541 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz"] Oct 31 17:45:33.034841 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:33.034811 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e8ccc4b-swntz"] Oct 31 17:45:34.372914 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:34.372872 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d22b3c-64fb-4916-9448-a58a5fe8de9c" path="/var/lib/kubelet/pods/e9d22b3c-64fb-4916-9448-a58a5fe8de9c/volumes" Oct 31 17:45:38.571934 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:45:38.571897 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:45:48.644986 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:48.644947 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:45:58.872978 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:45:58.872938 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:46:00.130122 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:00.130089 7618 scope.go:122] "RemoveContainer" containerID="2e76e4ceaa439833d8d1fc48df718fa2314c2155ed63969d4611f31ae3f03070" Oct 31 17:46:00.141275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:00.141237 7618 scope.go:122] "RemoveContainer" containerID="d612dd3653cd4e4361ef87de5e52c497c38fe25df6dbb11f822195086e629d92" Oct 31 17:46:00.156863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:00.156837 7618 scope.go:122] "RemoveContainer" containerID="0bc13eea3db8e1a32b2630980a673b6755dc650131a1e1ab58a2c0e1ebe4ad10" Oct 31 17:46:00.165555 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:00.165527 7618 scope.go:122] "RemoveContainer" containerID="d76f9a0eb4234d7bd7d3fdd56e0056a3946eb75ba88fd765bfd34386bfd1a82c" Oct 31 17:46:00.174756 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:00.174733 7618 scope.go:122] "RemoveContainer" containerID="f994158955406d91c1df1b49cd140b8a82e10ec1b45303cb21382ae121d2a765" Oct 31 17:46:00.183442 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:00.183423 7618 scope.go:122] "RemoveContainer" containerID="cd1572290d0e02d98b29ee7f367c48950963d8696739c671f38a9ddaceafa43b" Oct 31 17:46:03.471503 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:03.471456 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:46:08.571803 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:46:08.571764 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:46:15.023244 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:15.023149 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp"] Oct 31 17:46:15.033151 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:15.033121 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-200c6dfe-8v5qp"] Oct 31 17:46:16.373347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:16.373303 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291" path="/var/lib/kubelet/pods/dc1ee1dd-f558-4cfe-a2f9-81ebc3e63291/volumes" Oct 31 17:46:20.503356 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:20.503318 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:46:25.404377 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:25.404331 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:46:34.024133 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:34.024097 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd"] Oct 31 17:46:34.026418 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:34.026397 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644484c9-m57xd"] Oct 31 17:46:34.372652 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:34.372577 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495ff2b5-327a-424e-a596-9c2b3dbfaf74" path="/var/lib/kubelet/pods/495ff2b5-327a-424e-a596-9c2b3dbfaf74/volumes" Oct 31 17:46:38.574430 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:46:38.574391 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:46:40.878646 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:40.878607 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:46:45.032707 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:45.032666 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm"] Oct 31 17:46:45.042903 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:45.042870 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dc2860-qndhm"] Oct 31 17:46:45.777930 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:45.777896 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:46:46.377139 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:46:46.377086 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de3ccc3-7acf-451f-a5a9-905a474d1a68" path="/var/lib/kubelet/pods/0de3ccc3-7acf-451f-a5a9-905a474d1a68/volumes" Oct 31 17:47:00.230339 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:00.230302 7618 scope.go:122] "RemoveContainer" containerID="52863280df042351b2b84ae0574e6890d67d0626f63c6375fe64639e32ac90e3" Oct 31 17:47:01.029059 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:01.029023 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c"] Oct 31 17:47:01.039184 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:01.039159 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d83c999-2xm7c"] Oct 31 17:47:02.372886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:02.372847 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a06a816-c2a1-4e85-bf30-fc36ae674b7e" path="/var/lib/kubelet/pods/7a06a816-c2a1-4e85-bf30-fc36ae674b7e/volumes" Oct 31 17:47:03.881585 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:03.881549 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:47:08.641282 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:47:08.638766 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:47:08.755286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:08.753298 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:47:13.025297 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:13.025246 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts"] Oct 31 17:47:13.035436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:13.035405 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5219a4-p69ts"] Oct 31 17:47:14.373020 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:14.372983 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba87c0a-822f-467a-ab42-26cba2077357" path="/var/lib/kubelet/pods/8ba87c0a-822f-467a-ab42-26cba2077357/volumes" Oct 31 17:47:16.028590 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:16.028559 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx"] Oct 31 17:47:16.038870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:16.038838 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69af8050-vcrtx"] Oct 31 17:47:16.372737 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:16.372655 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f95a8b4-8da7-4205-9e22-6d041871af15" path="/var/lib/kubelet/pods/8f95a8b4-8da7-4205-9e22-6d041871af15/volumes" Oct 31 17:47:18.039095 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:18.039063 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz"] Oct 31 17:47:18.049443 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:18.049399 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6201fc8b-wg7bz"] Oct 31 17:47:18.372794 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:18.372714 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03356c26-fbaf-405d-9555-9fba69ce296f" path="/var/lib/kubelet/pods/03356c26-fbaf-405d-9555-9fba69ce296f/volumes" Oct 31 17:47:22.023115 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:22.023064 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67"] Oct 31 17:47:22.033423 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:22.033392 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a1b7134-snq67"] Oct 31 17:47:22.373093 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:22.373012 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7" path="/var/lib/kubelet/pods/e84d61c3-fa1d-46dc-9e72-1e5f58d05ff7/volumes" Oct 31 17:47:23.784722 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:23.784684 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:47:34.543700 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:34.543660 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:47:35.024407 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:35.024367 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp"] Oct 31 17:47:35.035616 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:35.035573 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10617ae3-rcvsp"] Oct 31 17:47:36.393514 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:36.393446 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4fbb38-4cfa-4aa6-9921-4516f153231f" path="/var/lib/kubelet/pods/ab4fbb38-4cfa-4aa6-9921-4516f153231f/volumes" Oct 31 17:47:37.144886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:37.144364 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kd6mw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:47:37.144886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:37.144448 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" podUID="df608734-a9f8-40c6-9430-1b25fa3389e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:47:38.577743 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:47:38.577697 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:47:39.666727 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:39.666544 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:47:47.023440 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:47.023400 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk"] Oct 31 17:47:47.033747 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:47.033718 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4b5981-k6cdk"] Oct 31 17:47:48.372540 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:48.372504 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c586ef82-a0d9-4dcb-8504-339d26c88b04" path="/var/lib/kubelet/pods/c586ef82-a0d9-4dcb-8504-339d26c88b04/volumes" Oct 31 17:47:49.031482 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:49.031443 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz"] Oct 31 17:47:49.041772 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:49.041745 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d81ce110-ghfjz"] Oct 31 17:47:50.026244 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:50.026195 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9"] Oct 31 17:47:50.036607 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:50.036575 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb6e677-g7bn9"] Oct 31 17:47:50.372275 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:50.372179 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3288d2-a16e-4770-9e3e-b06584d07dd8" path="/var/lib/kubelet/pods/9d3288d2-a16e-4770-9e3e-b06584d07dd8/volumes" Oct 31 17:47:50.372633 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:50.372619 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf57bc3-4558-446e-96bb-05d057e94b3c" path="/var/lib/kubelet/pods/dbf57bc3-4558-446e-96bb-05d057e94b3c/volumes" Oct 31 17:47:55.543886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:55.543847 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:47:59.137749 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:59.137714 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:47:59.141001 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:47:59.140973 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:48:00.253923 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:00.253880 7618 scope.go:122] "RemoveContainer" containerID="5c7a1d36acfc03b271d7f02591f37ad9b0438ab4b7d9715729cb7995c7a68983" Oct 31 17:48:00.263439 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:00.263407 7618 scope.go:122] "RemoveContainer" containerID="20637b2690dd6c202d415338d2ca53c03e1e7dbb8d799c59bd15f394e1cede8a" Oct 31 17:48:00.276106 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:00.276075 7618 scope.go:122] "RemoveContainer" containerID="b7a1c1d01617110cbd7e083bd7367785545012bcb2f8b7845c6ee35916e2394c" Oct 31 17:48:00.285287 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:00.285241 7618 scope.go:122] "RemoveContainer" containerID="be134a2d68c48bf0cb84a9595b597122db77498aa139512c2239f48532b6deef" Oct 31 17:48:00.293688 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:00.293663 7618 scope.go:122] "RemoveContainer" containerID="77a2b14e1e1d04e099cd1e6b1a8425960d2db0497f9a928223f6f00d8972fc8a" Oct 31 17:48:00.302595 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:00.302576 7618 scope.go:122] "RemoveContainer" containerID="bebec3cf81d61f9987e95b7d5f07cb1c352704a8e4dd768fd402bcb6ffc6b229" Oct 31 17:48:00.387882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:00.387833 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:48:08.572006 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:48:08.571958 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:48:09.261364 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:09.261320 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:48:27.066124 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:27.066076 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:48:37.467436 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:37.467397 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:48:38.571395 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:48:38.571356 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:48:56.853918 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:56.853874 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:48:58.519552 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:58.519501 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-hk9mq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:48:58.521579 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:48:58.521539 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-hk9mq" podUID="f04cec43-eec4-4872-8f9b-8609fc89a551" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:49:00.351352 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:00.351315 7618 scope.go:122] "RemoveContainer" containerID="b47e26d9bcc2ba03d671959f0893e17f2c2a72097f5fc73bf206ca165b218e29" Oct 31 17:49:00.361111 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:00.361082 7618 scope.go:122] "RemoveContainer" containerID="fb93d62c1649d8a86201af24b5a407d4e338e07e043ef983c9b46d0bcdb9cf40" Oct 31 17:49:00.370159 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:00.370134 7618 scope.go:122] "RemoveContainer" containerID="d794f304560ec837e9fe50166048d6a5bf2cd6aeeb3e791b00496b62001be5fb" Oct 31 17:49:00.378467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:00.378446 7618 scope.go:122] "RemoveContainer" containerID="fa0b5f048055a0723ca29348269ba122d343b6337d929ff8d98b7b04ed71a6df" Oct 31 17:49:00.386458 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:00.386439 7618 scope.go:122] "RemoveContainer" containerID="fcd00a5f5e84d3729dd1b0e002cf2cbd47f794513567ff1e92b1b244578b5add" Oct 31 17:49:00.394216 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:00.394196 7618 scope.go:122] "RemoveContainer" containerID="fcd672852127230b9d808f460265d1b8cfa9d925c320cdd4c0e748523484667d" Oct 31 17:49:00.404424 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:00.404400 7618 scope.go:122] "RemoveContainer" containerID="edd4ebf31beaeacb6d5ef3f8cd3f20556e351062697fdaf001d6c042f8cae617" Oct 31 17:49:00.412449 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:00.412422 7618 scope.go:122] "RemoveContainer" containerID="fb54c822ac096aaa50ccd3ca1d70e970c1493a8cd2d72edd1b827fdffc5d9dc1" Oct 31 17:49:00.421529 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:00.421505 7618 scope.go:122] "RemoveContainer" containerID="49ca131219313165d7f092acf44dd40ff01b2c7b36f86950419b6f4769b9fb94" Oct 31 17:49:00.429087 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:00.429066 7618 scope.go:122] "RemoveContainer" containerID="e58c4c2e712fb4337a50d19451140cde7ac8e47b76dce64717f2bd1cd2f1e881" Oct 31 17:49:00.436770 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:00.436746 7618 scope.go:122] "RemoveContainer" containerID="47ec1fc59636c5edda596c6fea43a626b1e54fb2f09a136429af60a561be59c0" Oct 31 17:49:02.375855 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:02.375824 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:49:08.572413 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:49:08.572374 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:49:17.065377 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:17.065279 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:49:30.997201 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:30.997163 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:49:38.572586 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:49:38.572540 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:49:46.472961 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:49:46.472922 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:50:08.572400 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:50:08.572353 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:50:10.022507 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:10.022471 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn"] Oct 31 17:50:10.032592 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:10.032556 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ldcpn"] Oct 31 17:50:10.373415 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:10.373338 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc4369c-2baf-495d-84c1-b444f527bfbc" path="/var/lib/kubelet/pods/8dc4369c-2baf-495d-84c1-b444f527bfbc/volumes" Oct 31 17:50:11.560478 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:11.560440 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:50:25.693009 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:25.692971 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:50:38.572054 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:50:38.572008 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:50:44.075012 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:44.074970 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:50:49.969347 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:49.969312 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:50:55.988546 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:55.988502 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q"] Oct 31 17:50:55.989058 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:55.988755 7618 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" podUID="45a7a2b9-c43d-47f7-a2ee-fb0575d7accb" containerName="manager" containerID="cri-o://a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9" gracePeriod=10 Oct 31 17:50:56.247926 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.247860 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" Oct 31 17:50:56.252023 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.251992 7618 generic.go:356] "Generic (PLEG): container finished" podID="45a7a2b9-c43d-47f7-a2ee-fb0575d7accb" containerID="a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9" exitCode=0 Oct 31 17:50:56.252144 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.252047 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" Oct 31 17:50:56.252144 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.252077 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" event={"ID":"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb","Type":"ContainerDied","Data":"a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9"} Oct 31 17:50:56.252144 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.252128 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q" event={"ID":"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb","Type":"ContainerDied","Data":"f2a2d0d5e9f66a0798469fdae46c6396d7732473c3706ac22b3be3ff3c235e69"} Oct 31 17:50:56.252244 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.252154 7618 scope.go:122] "RemoveContainer" containerID="a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9" Oct 31 17:50:56.262361 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.262342 7618 scope.go:122] "RemoveContainer" containerID="a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9" Oct 31 17:50:56.262642 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:50:56.262620 7618 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9\": container with ID starting with a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9 not found: ID does not exist" containerID="a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9" Oct 31 17:50:56.262716 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.262655 7618 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9"} err="failed to get container status \"a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9\": rpc error: code = NotFound desc = could not find container \"a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9\": container with ID starting with a811fa990ba63269a31e93e7ab36b92a9965e0b7b83d13f60131b10ae80e1df9 not found: ID does not exist" Oct 31 17:50:56.333906 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.333874 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-kube-api-access-5znd9\" (UniqueName: \"kubernetes.io/projected/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-kube-api-access-5znd9\") pod \"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb\" (UID: \"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb\") " Oct 31 17:50:56.334144 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.333923 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-extensions-socket-volume\") pod \"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb\" (UID: \"45a7a2b9-c43d-47f7-a2ee-fb0575d7accb\") " Oct 31 17:50:56.334423 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.334388 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-extensions-socket-volume" pod "45a7a2b9-c43d-47f7-a2ee-fb0575d7accb" (UID: "45a7a2b9-c43d-47f7-a2ee-fb0575d7accb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 17:50:56.336445 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.336414 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-kube-api-access-5znd9" pod "45a7a2b9-c43d-47f7-a2ee-fb0575d7accb" (UID: "45a7a2b9-c43d-47f7-a2ee-fb0575d7accb"). InnerVolumeSpecName "kube-api-access-5znd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:50:56.436113 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.436074 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5znd9\" (UniqueName: \"kubernetes.io/projected/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-kube-api-access-5znd9\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:50:56.436113 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.436111 7618 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb-extensions-socket-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:50:56.586360 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.586269 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q"] Oct 31 17:50:56.589860 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:56.589835 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6hs6q"] Oct 31 17:50:58.372230 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:50:58.372193 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a7a2b9-c43d-47f7-a2ee-fb0575d7accb" path="/var/lib/kubelet/pods/45a7a2b9-c43d-47f7-a2ee-fb0575d7accb/volumes" Oct 31 17:51:08.571372 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:51:08.571320 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:51:38.570877 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:51:38.570836 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:51:59.122357 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.122314 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf"] Oct 31 17:51:59.122784 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.122747 7618 cpu_manager.go:414] "containerMap: removing container" podUID="8dc4369c-2baf-495d-84c1-b444f527bfbc" containerName="revoke-keys" Oct 31 17:51:59.122784 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.122760 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc4369c-2baf-495d-84c1-b444f527bfbc" containerName="revoke-keys" Oct 31 17:51:59.122784 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.122778 7618 cpu_manager.go:414] "containerMap: removing container" podUID="45a7a2b9-c43d-47f7-a2ee-fb0575d7accb" containerName="manager" Oct 31 17:51:59.122784 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.122785 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a7a2b9-c43d-47f7-a2ee-fb0575d7accb" containerName="manager" Oct 31 17:51:59.122920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.122858 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc4369c-2baf-495d-84c1-b444f527bfbc" containerName="revoke-keys" Oct 31 17:51:59.122920 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.122868 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a7a2b9-c43d-47f7-a2ee-fb0575d7accb" containerName="manager" Oct 31 17:51:59.126074 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.126055 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" Oct 31 17:51:59.148953 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.148913 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4m7zn\"" Oct 31 17:51:59.150491 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.150465 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf"] Oct 31 17:51:59.159404 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.159375 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/893148fa-6f42-4363-ada7-0108932094f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5jgkf\" (UID: \"893148fa-6f42-4363-ada7-0108932094f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" Oct 31 17:51:59.159567 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.159444 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cw8q\" (UniqueName: \"kubernetes.io/projected/893148fa-6f42-4363-ada7-0108932094f2-kube-api-access-2cw8q\") pod \"kuadrant-operator-controller-manager-678d87766f-5jgkf\" (UID: \"893148fa-6f42-4363-ada7-0108932094f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" Oct 31 17:51:59.260755 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.260708 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/893148fa-6f42-4363-ada7-0108932094f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5jgkf\" (UID: \"893148fa-6f42-4363-ada7-0108932094f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" Oct 31 17:51:59.260934 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.260881 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cw8q\" (UniqueName: \"kubernetes.io/projected/893148fa-6f42-4363-ada7-0108932094f2-kube-api-access-2cw8q\") pod \"kuadrant-operator-controller-manager-678d87766f-5jgkf\" (UID: \"893148fa-6f42-4363-ada7-0108932094f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" Oct 31 17:51:59.261117 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.261098 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/893148fa-6f42-4363-ada7-0108932094f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5jgkf\" (UID: \"893148fa-6f42-4363-ada7-0108932094f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" Oct 31 17:51:59.278376 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.278348 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cw8q\" (UniqueName: \"kubernetes.io/projected/893148fa-6f42-4363-ada7-0108932094f2-kube-api-access-2cw8q\") pod \"kuadrant-operator-controller-manager-678d87766f-5jgkf\" (UID: \"893148fa-6f42-4363-ada7-0108932094f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" Oct 31 17:51:59.439557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.439524 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" Oct 31 17:51:59.761035 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:51:59.760945 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf"] Oct 31 17:51:59.765462 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:51:59.765429 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893148fa_6f42_4363_ada7_0108932094f2.slice/crio-e4a57ad24fa1145d70cd8fb490b9f7acd3d3b5f15b8d7b27936e5a1f1421c4f1 WatchSource:0}: Error finding container e4a57ad24fa1145d70cd8fb490b9f7acd3d3b5f15b8d7b27936e5a1f1421c4f1: Status 404 returned error can't find the container with id e4a57ad24fa1145d70cd8fb490b9f7acd3d3b5f15b8d7b27936e5a1f1421c4f1 Oct 31 17:52:00.519585 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:00.519549 7618 scope.go:122] "RemoveContainer" containerID="ee78a0df0e3471fa71f3139ffe71091edc5428fe23c659525f148c8e6f91c901" Oct 31 17:52:00.556865 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:00.556823 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" event={"ID":"893148fa-6f42-4363-ada7-0108932094f2","Type":"ContainerStarted","Data":"9c6f621da7939477528f8e93b13ffeabc87f1989ffd3224cca444011f8cb3506"} Oct 31 17:52:00.557026 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:00.556886 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" event={"ID":"893148fa-6f42-4363-ada7-0108932094f2","Type":"ContainerStarted","Data":"e4a57ad24fa1145d70cd8fb490b9f7acd3d3b5f15b8d7b27936e5a1f1421c4f1"} Oct 31 17:52:00.557026 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:00.556911 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" Oct 31 17:52:00.614889 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:00.614834 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" podStartSLOduration=1.6148150289999998 podStartE2EDuration="1.614815029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:52:00.604152775 +0000 UTC m=+1442.352127241" watchObservedRunningTime="2026-10-31 17:52:00.614815029 +0000 UTC m=+1442.362789497" Oct 31 17:52:08.571409 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:52:08.571367 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:52:11.565114 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:11.565032 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" Oct 31 17:52:17.669235 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:17.669197 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:52:32.388639 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:32.388605 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:52:37.147467 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:37.147405 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kd6mw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:52:37.148161 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:37.148124 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kd6mw" podUID="df608734-a9f8-40c6-9430-1b25fa3389e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:52:37.489332 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:37.489149 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:52:38.572967 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:52:38.572929 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:52:53.466076 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:53.466036 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:52:59.175681 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:59.175652 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:52:59.184444 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:52:59.184416 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:53:03.392940 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:53:03.392899 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:53:08.572032 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:53:08.571992 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:53:16.380055 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:53:16.380018 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:53:33.991943 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:53:33.991902 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:53:38.576977 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:53:38.576931 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:53:58.168269 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:53:58.168223 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:54:08.571237 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:54:08.571197 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:54:11.772284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:54:11.772224 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:54:23.368071 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:54:23.368028 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:54:38.581364 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:54:38.581318 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:54:39.409048 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:54:39.409010 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:54:42.384278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:54:42.383344 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4vwr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:54:42.384278 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:54:42.383415 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" podUID="e63bb320-851c-4bd6-880b-906a3c12bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:55:00.549188 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:55:00.549107 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:55:08.571370 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:55:08.571334 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:55:14.653446 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:55:14.653363 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:55:21.594647 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:55:21.594604 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:55:29.676030 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:55:29.675985 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:55:36.753833 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:55:36.753793 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:55:38.571395 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:55:38.571353 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:55:52.989944 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:55:52.989899 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:56:08.571734 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:56:08.571688 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:56:09.571777 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:09.571735 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:56:19.809453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:19.809412 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs"] Oct 31 17:56:19.813524 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:19.813503 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:19.838223 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:19.838194 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 17:56:19.838401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:19.838232 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:56:19.838401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:19.838239 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mr9x2\"" Oct 31 17:56:19.838401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:19.838333 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 17:56:19.839803 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:19.839784 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs"] Oct 31 17:56:19.938056 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:19.938022 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d82e915-f88vs\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:19.938288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:19.938139 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfvl8\" (UniqueName: \"kubernetes.io/projected/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-kube-api-access-lfvl8\") pod \"maas-api-revoke-keys-e2e-ait-5d82e915-f88vs\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:19.938288 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:19.938185 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d82e915-f88vs\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:20.039704 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:20.039667 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d82e915-f88vs\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:20.039890 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:20.039737 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfvl8\" (UniqueName: \"kubernetes.io/projected/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-kube-api-access-lfvl8\") pod \"maas-api-revoke-keys-e2e-ait-5d82e915-f88vs\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:20.039890 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:20.039775 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d82e915-f88vs\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:20.040523 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:20.040503 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d82e915-f88vs\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:20.042030 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:20.042013 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5d82e915-f88vs\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:20.063897 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:56:20.063817 7618 projected.go:196] Error preparing data for projected volume kube-api-access-lfvl8 for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 17:56:20.063897 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:56:20.063891 7618 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-kube-api-access-lfvl8 podName:c183c57a-6047-4a2a-9a7d-60de1a6ebc90 nodeName:}" failed. No retries permitted until 2026-10-31 17:56:20.563871211 +0000 UTC m=+1702.311845671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lfvl8" (UniqueName: "kubernetes.io/projected/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-kube-api-access-lfvl8") pod "maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" (UID: "c183c57a-6047-4a2a-9a7d-60de1a6ebc90") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 31 17:56:20.646135 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:20.646080 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfvl8\" (UniqueName: \"kubernetes.io/projected/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-kube-api-access-lfvl8\") pod \"maas-api-revoke-keys-e2e-ait-5d82e915-f88vs\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:20.663149 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:20.663115 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfvl8\" (UniqueName: \"kubernetes.io/projected/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-kube-api-access-lfvl8\") pod \"maas-api-revoke-keys-e2e-ait-5d82e915-f88vs\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:20.726427 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:20.726393 7618 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" secret="" err="secret \"maas-api-cleanup-dockercfg-mr9x2\" not found" Oct 31 17:56:20.726604 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:20.726447 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:21.019699 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:21.019573 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs"] Oct 31 17:56:21.023265 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:56:21.023226 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc183c57a_6047_4a2a_9a7d_60de1a6ebc90.slice/crio-f770fe6a37a6ab7f1db9397803bd870a0f387bf285acfc838402db3f44b59476 WatchSource:0}: Error finding container f770fe6a37a6ab7f1db9397803bd870a0f387bf285acfc838402db3f44b59476: Status 404 returned error can't find the container with id f770fe6a37a6ab7f1db9397803bd870a0f387bf285acfc838402db3f44b59476 Oct 31 17:56:21.032882 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:21.032855 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" event={"ID":"c183c57a-6047-4a2a-9a7d-60de1a6ebc90","Type":"ContainerStarted","Data":"f770fe6a37a6ab7f1db9397803bd870a0f387bf285acfc838402db3f44b59476"} Oct 31 17:56:22.039694 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:22.039659 7618 generic.go:356] "Generic (PLEG): container finished" podID="c183c57a-6047-4a2a-9a7d-60de1a6ebc90" containerID="48ec8988228f60e90aa904cd8978bc6b7ceb23eb216f01ad9c6e2a15517b317f" exitCode=0 Oct 31 17:56:22.040077 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:22.039716 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" event={"ID":"c183c57a-6047-4a2a-9a7d-60de1a6ebc90","Type":"ContainerDied","Data":"48ec8988228f60e90aa904cd8978bc6b7ceb23eb216f01ad9c6e2a15517b317f"} Oct 31 17:56:23.176646 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:23.176618 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:23.272867 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:23.272834 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-openshift-ca\") pod \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " Oct 31 17:56:23.273067 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:23.272927 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-kube-api-access-lfvl8\" (UniqueName: \"kubernetes.io/projected/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-kube-api-access-lfvl8\") pod \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " Oct 31 17:56:23.273067 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:23.273036 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-xks-ca\") pod \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\" (UID: \"c183c57a-6047-4a2a-9a7d-60de1a6ebc90\") " Oct 31 17:56:23.273328 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:23.273268 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-openshift-ca" pod "c183c57a-6047-4a2a-9a7d-60de1a6ebc90" (UID: "c183c57a-6047-4a2a-9a7d-60de1a6ebc90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:56:23.273432 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:23.273417 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:56:23.275072 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:23.275045 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-xks-ca" pod "c183c57a-6047-4a2a-9a7d-60de1a6ebc90" (UID: "c183c57a-6047-4a2a-9a7d-60de1a6ebc90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:56:23.275169 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:23.275116 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-kube-api-access-lfvl8" pod "c183c57a-6047-4a2a-9a7d-60de1a6ebc90" (UID: "c183c57a-6047-4a2a-9a7d-60de1a6ebc90"). InnerVolumeSpecName "kube-api-access-lfvl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:56:23.374451 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:23.374357 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:56:23.374451 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:23.374394 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfvl8\" (UniqueName: \"kubernetes.io/projected/c183c57a-6047-4a2a-9a7d-60de1a6ebc90-kube-api-access-lfvl8\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:56:24.050952 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:24.050915 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" Oct 31 17:56:24.050952 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:24.050932 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs" event={"ID":"c183c57a-6047-4a2a-9a7d-60de1a6ebc90","Type":"ContainerDied","Data":"f770fe6a37a6ab7f1db9397803bd870a0f387bf285acfc838402db3f44b59476"} Oct 31 17:56:24.051220 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:24.050983 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f770fe6a37a6ab7f1db9397803bd870a0f387bf285acfc838402db3f44b59476" Oct 31 17:56:33.056617 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:33.056579 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:56:33.613496 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:33.613459 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs"] Oct 31 17:56:33.624085 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:33.624039 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5d82e915-f88vs"] Oct 31 17:56:34.372663 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:34.372628 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c183c57a-6047-4a2a-9a7d-60de1a6ebc90" path="/var/lib/kubelet/pods/c183c57a-6047-4a2a-9a7d-60de1a6ebc90/volumes" Oct 31 17:56:38.571238 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:56:38.571201 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:56:46.376485 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:56:46.376439 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:57:08.571302 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:57:08.571231 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:57:38.571356 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:57:38.571313 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:57:59.215475 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:57:59.215447 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:57:59.223090 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:57:59.223065 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tdrsm_7a3fc242-f912-4cdd-8f92-dcd8319ba40a/ovn-acl-logging/0.log" Oct 31 17:58:08.570927 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:58:08.570888 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:58:18.565732 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.565643 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8"] Oct 31 17:58:18.566309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.566151 7618 cpu_manager.go:414] "containerMap: removing container" podUID="c183c57a-6047-4a2a-9a7d-60de1a6ebc90" containerName="revoke-keys" Oct 31 17:58:18.566309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.566165 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c183c57a-6047-4a2a-9a7d-60de1a6ebc90" containerName="revoke-keys" Oct 31 17:58:18.566309 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.566237 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="c183c57a-6047-4a2a-9a7d-60de1a6ebc90" containerName="revoke-keys" Oct 31 17:58:18.569438 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.569422 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:18.581977 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.581949 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 17:58:18.582098 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.581991 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 17:58:18.592517 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.592489 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:58:18.592660 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.592527 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxlqk\"" Oct 31 17:58:18.594522 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.594497 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8"] Oct 31 17:58:18.702626 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.702586 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgj5z\" (UniqueName: \"kubernetes.io/projected/3b1fd772-79a2-4896-bc84-732ad3ade9d1-kube-api-access-mgj5z\") pod \"maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8\" (UID: \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:18.702626 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.702633 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8\" (UID: \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:18.702845 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.702720 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8\" (UID: \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:18.803709 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.803672 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgj5z\" (UniqueName: \"kubernetes.io/projected/3b1fd772-79a2-4896-bc84-732ad3ade9d1-kube-api-access-mgj5z\") pod \"maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8\" (UID: \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:18.803886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.803725 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8\" (UID: \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:18.803886 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.803772 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8\" (UID: \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:18.804463 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.804427 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8\" (UID: \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:18.806315 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.806289 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8\" (UID: \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:18.820932 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.820868 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgj5z\" (UniqueName: \"kubernetes.io/projected/3b1fd772-79a2-4896-bc84-732ad3ade9d1-kube-api-access-mgj5z\") pod \"maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8\" (UID: \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:18.882589 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:18.882549 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:19.175502 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:19.175465 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8"] Oct 31 17:58:19.179182 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:58:19.179146 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1fd772_79a2_4896_bc84_732ad3ade9d1.slice/crio-2e00e5d7b0545f7cf3a6a4865088f5a246e289b0d58cbf70f4805d0422160d64 WatchSource:0}: Error finding container 2e00e5d7b0545f7cf3a6a4865088f5a246e289b0d58cbf70f4805d0422160d64: Status 404 returned error can't find the container with id 2e00e5d7b0545f7cf3a6a4865088f5a246e289b0d58cbf70f4805d0422160d64 Oct 31 17:58:19.639733 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:19.639693 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" event={"ID":"3b1fd772-79a2-4896-bc84-732ad3ade9d1","Type":"ContainerStarted","Data":"445d6b3cc06765418bc76d67b31f13bec752725bd9148d4bea827a23baaaaacd"} Oct 31 17:58:19.640141 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:19.639747 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" event={"ID":"3b1fd772-79a2-4896-bc84-732ad3ade9d1","Type":"ContainerStarted","Data":"2e00e5d7b0545f7cf3a6a4865088f5a246e289b0d58cbf70f4805d0422160d64"} Oct 31 17:58:19.673657 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:19.673605 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" podStartSLOduration=1.673591473 podStartE2EDuration="1.673591473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:58:19.672552644 +0000 UTC m=+1821.420527124" watchObservedRunningTime="2026-10-31 17:58:19.673591473 +0000 UTC m=+1821.421565939" Oct 31 17:58:27.682004 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:27.681965 7618 generic.go:356] "Generic (PLEG): container finished" podID="3b1fd772-79a2-4896-bc84-732ad3ade9d1" containerID="445d6b3cc06765418bc76d67b31f13bec752725bd9148d4bea827a23baaaaacd" exitCode=0 Oct 31 17:58:27.682533 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:27.682018 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" event={"ID":"3b1fd772-79a2-4896-bc84-732ad3ade9d1","Type":"ContainerDied","Data":"445d6b3cc06765418bc76d67b31f13bec752725bd9148d4bea827a23baaaaacd"} Oct 31 17:58:28.826394 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:28.826365 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:28.906969 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:28.906935 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-xks-ca\") pod \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\" (UID: \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\") " Oct 31 17:58:28.907150 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:28.907019 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1fd772-79a2-4896-bc84-732ad3ade9d1-kube-api-access-mgj5z\" (UniqueName: \"kubernetes.io/projected/3b1fd772-79a2-4896-bc84-732ad3ade9d1-kube-api-access-mgj5z\") pod \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\" (UID: \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\") " Oct 31 17:58:28.907150 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:28.907077 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-openshift-ca\") pod \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\" (UID: \"3b1fd772-79a2-4896-bc84-732ad3ade9d1\") " Oct 31 17:58:28.907509 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:28.907486 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-openshift-ca" pod "3b1fd772-79a2-4896-bc84-732ad3ade9d1" (UID: "3b1fd772-79a2-4896-bc84-732ad3ade9d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:58:28.909054 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:28.909022 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-xks-ca" pod "3b1fd772-79a2-4896-bc84-732ad3ade9d1" (UID: "3b1fd772-79a2-4896-bc84-732ad3ade9d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:58:28.909195 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:28.909174 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1fd772-79a2-4896-bc84-732ad3ade9d1-kube-api-access-mgj5z" pod "3b1fd772-79a2-4896-bc84-732ad3ade9d1" (UID: "3b1fd772-79a2-4896-bc84-732ad3ade9d1"). InnerVolumeSpecName "kube-api-access-mgj5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:58:29.007992 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:29.007899 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:58:29.007992 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:29.007938 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1fd772-79a2-4896-bc84-732ad3ade9d1-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:58:29.007992 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:29.007948 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgj5z\" (UniqueName: \"kubernetes.io/projected/3b1fd772-79a2-4896-bc84-732ad3ade9d1-kube-api-access-mgj5z\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:58:29.693377 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:29.693343 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" Oct 31 17:58:29.693557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:29.693373 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8" event={"ID":"3b1fd772-79a2-4896-bc84-732ad3ade9d1","Type":"ContainerDied","Data":"2e00e5d7b0545f7cf3a6a4865088f5a246e289b0d58cbf70f4805d0422160d64"} Oct 31 17:58:29.693557 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:29.693422 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e00e5d7b0545f7cf3a6a4865088f5a246e289b0d58cbf70f4805d0422160d64" Oct 31 17:58:38.572191 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:58:38.572151 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:58:42.761551 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:42.761515 7618 generic.go:356] "Generic (PLEG): container finished" podID="0672b07e-809a-443d-9dd8-6df70e1e5ddf" containerID="5679bdbec08db9a9640e999d8e76ddb9705d5dcab6e55c6ca190e3aa64b604b7" exitCode=1 Oct 31 17:58:42.762059 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:42.761593 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" event={"ID":"0672b07e-809a-443d-9dd8-6df70e1e5ddf","Type":"ContainerDied","Data":"5679bdbec08db9a9640e999d8e76ddb9705d5dcab6e55c6ca190e3aa64b604b7"} Oct 31 17:58:42.762130 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:42.762059 7618 scope.go:122] "RemoveContainer" containerID="5679bdbec08db9a9640e999d8e76ddb9705d5dcab6e55c6ca190e3aa64b604b7" Oct 31 17:58:42.762493 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:42.762473 7618 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 17:58:42.763647 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:42.763355 7618 generic.go:356] "Generic (PLEG): container finished" podID="893148fa-6f42-4363-ada7-0108932094f2" containerID="9c6f621da7939477528f8e93b13ffeabc87f1989ffd3224cca444011f8cb3506" exitCode=0 Oct 31 17:58:42.763647 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:42.763420 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" event={"ID":"893148fa-6f42-4363-ada7-0108932094f2","Type":"ContainerDied","Data":"9c6f621da7939477528f8e93b13ffeabc87f1989ffd3224cca444011f8cb3506"} Oct 31 17:58:42.763747 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:42.763732 7618 scope.go:122] "RemoveContainer" containerID="9c6f621da7939477528f8e93b13ffeabc87f1989ffd3224cca444011f8cb3506" Oct 31 17:58:43.770566 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:43.770523 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" event={"ID":"0672b07e-809a-443d-9dd8-6df70e1e5ddf","Type":"ContainerStarted","Data":"986d436fcd1a482531bcbd036cc88901a0f386772bce06ba7e247c293d188899"} Oct 31 17:58:43.771038 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:43.770755 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" Oct 31 17:58:43.772770 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:43.772744 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" event={"ID":"893148fa-6f42-4363-ada7-0108932094f2","Type":"ContainerStarted","Data":"3f29a452717ac6e5646b373866a0402c73851a30899150eea78ddb1383a27a21"} Oct 31 17:58:43.772937 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:43.772921 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" Oct 31 17:58:46.640219 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:46.640184 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8"] Oct 31 17:58:46.650647 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:46.650545 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-139d26-244f8"] Oct 31 17:58:48.372319 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:48.372276 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1fd772-79a2-4896-bc84-732ad3ade9d1" path="/var/lib/kubelet/pods/3b1fd772-79a2-4896-bc84-732ad3ade9d1/volumes" Oct 31 17:58:54.780362 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:58:54.780327 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5jgkf" Oct 31 17:59:01.261403 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:01.261367 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:59:08.571901 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:59:08.571859 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:59:09.762246 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:09.762211 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:59:12.387284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:12.384407 7618 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-4vwr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:59:12.387284 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:12.384474 7618 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-4vwr9" podUID="e63bb320-851c-4bd6-880b-906a3c12bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:59:14.779069 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:14.779034 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2t6x8" Oct 31 17:59:16.398993 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.398960 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8"] Oct 31 17:59:16.399453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.399420 7618 cpu_manager.go:414] "containerMap: removing container" podUID="3b1fd772-79a2-4896-bc84-732ad3ade9d1" containerName="revoke-keys" Oct 31 17:59:16.399453 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.399432 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1fd772-79a2-4896-bc84-732ad3ade9d1" containerName="revoke-keys" Oct 31 17:59:16.399537 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.399517 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1fd772-79a2-4896-bc84-732ad3ade9d1" containerName="revoke-keys" Oct 31 17:59:16.403023 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.403004 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:16.419242 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.419212 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8"] Oct 31 17:59:16.427900 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.427874 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 17:59:16.428065 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.427917 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 17:59:16.428065 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.427900 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:59:16.428065 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.427950 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxlqk\"" Oct 31 17:59:16.432401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.432379 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-gbnh8\" (UID: \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:16.432458 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.432420 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-gbnh8\" (UID: \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:16.432599 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.432582 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2g4\" (UniqueName: \"kubernetes.io/projected/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-kube-api-access-2k2g4\") pod \"maas-api-revoke-sub-f03629db-gbnh8\" (UID: \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:16.533433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.533387 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-gbnh8\" (UID: \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:16.533433 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.533437 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-gbnh8\" (UID: \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:16.533674 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.533525 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2g4\" (UniqueName: \"kubernetes.io/projected/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-kube-api-access-2k2g4\") pod \"maas-api-revoke-sub-f03629db-gbnh8\" (UID: \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:16.534079 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.534052 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-gbnh8\" (UID: \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:16.535781 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.535756 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-gbnh8\" (UID: \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:16.550866 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.550832 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2g4\" (UniqueName: \"kubernetes.io/projected/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-kube-api-access-2k2g4\") pod \"maas-api-revoke-sub-f03629db-gbnh8\" (UID: \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:16.715817 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:16.715732 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:17.019082 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:17.018995 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8"] Oct 31 17:59:17.022797 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:59:17.022760 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf61d6f8_c549_4e80_b4bb_18d2e11f8fc0.slice/crio-2b18e60a867e30c4fe804f0dfbe84b5414793f44165a6106c055f62b81e7d373 WatchSource:0}: Error finding container 2b18e60a867e30c4fe804f0dfbe84b5414793f44165a6106c055f62b81e7d373: Status 404 returned error can't find the container with id 2b18e60a867e30c4fe804f0dfbe84b5414793f44165a6106c055f62b81e7d373 Oct 31 17:59:17.964896 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:17.964865 7618 generic.go:356] "Generic (PLEG): container finished" podID="df61d6f8-c549-4e80-b4bb-18d2e11f8fc0" containerID="b9afb9f0c687d54f72e04b7fc23e4ac56c03991759f12c7fdc42e9e80aadca36" exitCode=0 Oct 31 17:59:17.965296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:17.964947 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" event={"ID":"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0","Type":"ContainerDied","Data":"b9afb9f0c687d54f72e04b7fc23e4ac56c03991759f12c7fdc42e9e80aadca36"} Oct 31 17:59:17.965296 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:17.965001 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" event={"ID":"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0","Type":"ContainerStarted","Data":"2b18e60a867e30c4fe804f0dfbe84b5414793f44165a6106c055f62b81e7d373"} Oct 31 17:59:19.107459 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.107430 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:19.162273 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.162216 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-xks-ca\") pod \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\" (UID: \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\") " Oct 31 17:59:19.162449 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.162309 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-openshift-ca\") pod \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\" (UID: \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\") " Oct 31 17:59:19.162449 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.162370 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-kube-api-access-2k2g4\" (UniqueName: \"kubernetes.io/projected/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-kube-api-access-2k2g4\") pod \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\" (UID: \"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0\") " Oct 31 17:59:19.162722 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.162697 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-openshift-ca" pod "df61d6f8-c549-4e80-b4bb-18d2e11f8fc0" (UID: "df61d6f8-c549-4e80-b4bb-18d2e11f8fc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:59:19.162821 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.162804 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:59:19.165001 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.164970 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-xks-ca" pod "df61d6f8-c549-4e80-b4bb-18d2e11f8fc0" (UID: "df61d6f8-c549-4e80-b4bb-18d2e11f8fc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:59:19.165131 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.165085 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-kube-api-access-2k2g4" pod "df61d6f8-c549-4e80-b4bb-18d2e11f8fc0" (UID: "df61d6f8-c549-4e80-b4bb-18d2e11f8fc0"). InnerVolumeSpecName "kube-api-access-2k2g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:59:19.264000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.263901 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:59:19.264000 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.263941 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k2g4\" (UniqueName: \"kubernetes.io/projected/df61d6f8-c549-4e80-b4bb-18d2e11f8fc0-kube-api-access-2k2g4\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:59:19.975870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.975839 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" Oct 31 17:59:19.976057 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.975869 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gbnh8" event={"ID":"df61d6f8-c549-4e80-b4bb-18d2e11f8fc0","Type":"ContainerDied","Data":"2b18e60a867e30c4fe804f0dfbe84b5414793f44165a6106c055f62b81e7d373"} Oct 31 17:59:19.976057 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:19.975910 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b18e60a867e30c4fe804f0dfbe84b5414793f44165a6106c055f62b81e7d373" Oct 31 17:59:24.378413 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:24.378375 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 17:59:27.466740 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.466698 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r"] Oct 31 17:59:27.467474 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.467401 7618 cpu_manager.go:414] "containerMap: removing container" podUID="df61d6f8-c549-4e80-b4bb-18d2e11f8fc0" containerName="revoke-keys" Oct 31 17:59:27.467474 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.467423 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df61d6f8-c549-4e80-b4bb-18d2e11f8fc0" containerName="revoke-keys" Oct 31 17:59:27.467622 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.467592 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="df61d6f8-c549-4e80-b4bb-18d2e11f8fc0" containerName="revoke-keys" Oct 31 17:59:27.471242 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.471215 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:27.488107 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.488069 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r"] Oct 31 17:59:27.498795 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.498758 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:59:27.499235 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.499208 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 17:59:27.499401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.499311 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 17:59:27.499401 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.499368 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxlqk\"" Oct 31 17:59:27.552655 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.552617 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6fg2r\" (UID: \"630e7556-9a56-4ee8-a565-0dd5167ea24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:27.552848 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.552684 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8lsj\" (UniqueName: \"kubernetes.io/projected/630e7556-9a56-4ee8-a565-0dd5167ea24a-kube-api-access-l8lsj\") pod \"maas-api-revoke-sub-0165d7d2-6fg2r\" (UID: \"630e7556-9a56-4ee8-a565-0dd5167ea24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:27.552848 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.552808 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6fg2r\" (UID: \"630e7556-9a56-4ee8-a565-0dd5167ea24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:27.653988 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.653932 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6fg2r\" (UID: \"630e7556-9a56-4ee8-a565-0dd5167ea24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:27.654194 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.654002 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8lsj\" (UniqueName: \"kubernetes.io/projected/630e7556-9a56-4ee8-a565-0dd5167ea24a-kube-api-access-l8lsj\") pod \"maas-api-revoke-sub-0165d7d2-6fg2r\" (UID: \"630e7556-9a56-4ee8-a565-0dd5167ea24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:27.654194 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.654049 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6fg2r\" (UID: \"630e7556-9a56-4ee8-a565-0dd5167ea24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:27.654838 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.654803 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6fg2r\" (UID: \"630e7556-9a56-4ee8-a565-0dd5167ea24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:27.656799 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.656766 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6fg2r\" (UID: \"630e7556-9a56-4ee8-a565-0dd5167ea24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:27.671069 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.671041 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8lsj\" (UniqueName: \"kubernetes.io/projected/630e7556-9a56-4ee8-a565-0dd5167ea24a-kube-api-access-l8lsj\") pod \"maas-api-revoke-sub-0165d7d2-6fg2r\" (UID: \"630e7556-9a56-4ee8-a565-0dd5167ea24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:27.791512 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:27.791402 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:28.282276 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:28.261104 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r"] Oct 31 17:59:29.154619 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:29.154577 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" event={"ID":"630e7556-9a56-4ee8-a565-0dd5167ea24a","Type":"ContainerStarted","Data":"c31825b224d99e4ecd63f454140a0888329ea06088fe37e8a1713a08b3b4db28"} Oct 31 17:59:29.155286 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:29.155223 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" event={"ID":"630e7556-9a56-4ee8-a565-0dd5167ea24a","Type":"ContainerStarted","Data":"9a87b7cdabc18db154de77c302314f9160135b59f76d315c8171380e7a60a358"} Oct 31 17:59:29.199896 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:29.199820 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" podStartSLOduration=2.199799208 podStartE2EDuration="2.199799208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:59:29.196169678 +0000 UTC m=+1890.944144151" watchObservedRunningTime="2026-10-31 17:59:29.199799208 +0000 UTC m=+1890.947773675" Oct 31 17:59:30.161514 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:30.161474 7618 generic.go:356] "Generic (PLEG): container finished" podID="28290dd1-c75b-4c13-88ab-2f9a6dea4795" containerID="b18d15a17c6620fd1cd0fe118170b2f655771533e86e265c94291d2d1779371c" exitCode=1 Oct 31 17:59:30.161958 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:30.161543 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" event={"ID":"28290dd1-c75b-4c13-88ab-2f9a6dea4795","Type":"ContainerDied","Data":"b18d15a17c6620fd1cd0fe118170b2f655771533e86e265c94291d2d1779371c"} Oct 31 17:59:30.162036 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:30.161988 7618 scope.go:122] "RemoveContainer" containerID="b18d15a17c6620fd1cd0fe118170b2f655771533e86e265c94291d2d1779371c" Oct 31 17:59:30.163427 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:30.163271 7618 generic.go:356] "Generic (PLEG): container finished" podID="630e7556-9a56-4ee8-a565-0dd5167ea24a" containerID="c31825b224d99e4ecd63f454140a0888329ea06088fe37e8a1713a08b3b4db28" exitCode=0 Oct 31 17:59:30.163427 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:30.163334 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" event={"ID":"630e7556-9a56-4ee8-a565-0dd5167ea24a","Type":"ContainerDied","Data":"c31825b224d99e4ecd63f454140a0888329ea06088fe37e8a1713a08b3b4db28"} Oct 31 17:59:30.765628 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:30.765551 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:59:31.096127 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.096032 7618 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:59:31.170423 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.170382 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" event={"ID":"28290dd1-c75b-4c13-88ab-2f9a6dea4795","Type":"ContainerStarted","Data":"c928dda016bda41b05fe14c8c6c38e0d9fd384a2da5a333c41b472541817fa37"} Oct 31 17:59:31.170901 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.170626 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 17:59:31.308729 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.308701 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:31.376966 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.376887 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-xks-ca\") pod \"630e7556-9a56-4ee8-a565-0dd5167ea24a\" (UID: \"630e7556-9a56-4ee8-a565-0dd5167ea24a\") " Oct 31 17:59:31.378866 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.378841 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-xks-ca" pod "630e7556-9a56-4ee8-a565-0dd5167ea24a" (UID: "630e7556-9a56-4ee8-a565-0dd5167ea24a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:59:31.478114 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.478079 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630e7556-9a56-4ee8-a565-0dd5167ea24a-kube-api-access-l8lsj\" (UniqueName: \"kubernetes.io/projected/630e7556-9a56-4ee8-a565-0dd5167ea24a-kube-api-access-l8lsj\") pod \"630e7556-9a56-4ee8-a565-0dd5167ea24a\" (UID: \"630e7556-9a56-4ee8-a565-0dd5167ea24a\") " Oct 31 17:59:31.478281 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.478123 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-openshift-ca\") pod \"630e7556-9a56-4ee8-a565-0dd5167ea24a\" (UID: \"630e7556-9a56-4ee8-a565-0dd5167ea24a\") " Oct 31 17:59:31.478546 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.478525 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:59:31.478585 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.478553 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-openshift-ca" pod "630e7556-9a56-4ee8-a565-0dd5167ea24a" (UID: "630e7556-9a56-4ee8-a565-0dd5167ea24a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:59:31.480123 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.480104 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630e7556-9a56-4ee8-a565-0dd5167ea24a-kube-api-access-l8lsj" pod "630e7556-9a56-4ee8-a565-0dd5167ea24a" (UID: "630e7556-9a56-4ee8-a565-0dd5167ea24a"). InnerVolumeSpecName "kube-api-access-l8lsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:59:31.579760 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.579710 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8lsj\" (UniqueName: \"kubernetes.io/projected/630e7556-9a56-4ee8-a565-0dd5167ea24a-kube-api-access-l8lsj\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:59:31.579760 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:31.579755 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630e7556-9a56-4ee8-a565-0dd5167ea24a-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:59:32.176223 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:32.176192 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" Oct 31 17:59:32.176669 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:32.176232 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6fg2r" event={"ID":"630e7556-9a56-4ee8-a565-0dd5167ea24a","Type":"ContainerDied","Data":"9a87b7cdabc18db154de77c302314f9160135b59f76d315c8171380e7a60a358"} Oct 31 17:59:32.176669 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:32.176296 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a87b7cdabc18db154de77c302314f9160135b59f76d315c8171380e7a60a358" Oct 31 17:59:38.388955 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.388921 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr"] Oct 31 17:59:38.389404 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.389389 7618 cpu_manager.go:414] "containerMap: removing container" podUID="630e7556-9a56-4ee8-a565-0dd5167ea24a" containerName="revoke-keys" Oct 31 17:59:38.389458 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.389406 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630e7556-9a56-4ee8-a565-0dd5167ea24a" containerName="revoke-keys" Oct 31 17:59:38.389494 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.389487 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="630e7556-9a56-4ee8-a565-0dd5167ea24a" containerName="revoke-keys" Oct 31 17:59:38.392821 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.392798 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:38.417870 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.417833 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxlqk\"" Oct 31 17:59:38.418103 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.417876 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 17:59:38.418103 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.418031 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 17:59:38.418103 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.418049 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 17:59:38.419823 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.419801 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr"] Oct 31 17:59:38.450609 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.450565 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-s5zdr\" (UID: \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:38.450983 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.450955 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-s5zdr\" (UID: \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:38.451097 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.451075 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkn6\" (UniqueName: \"kubernetes.io/projected/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-kube-api-access-dzkn6\") pod \"maas-api-revoke-sub-6b19338f-s5zdr\" (UID: \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:38.552645 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.552589 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-s5zdr\" (UID: \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:38.552836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.552747 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-s5zdr\" (UID: \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:38.552836 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.552826 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkn6\" (UniqueName: \"kubernetes.io/projected/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-kube-api-access-dzkn6\") pod \"maas-api-revoke-sub-6b19338f-s5zdr\" (UID: \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:38.553529 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.553500 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-s5zdr\" (UID: \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:38.555715 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.555681 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-s5zdr\" (UID: \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:38.570823 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.570786 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkn6\" (UniqueName: \"kubernetes.io/projected/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-kube-api-access-dzkn6\") pod \"maas-api-revoke-sub-6b19338f-s5zdr\" (UID: \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:38.573190 ip-10-0-139-107 kubenswrapper[7618]: E1031 17:59:38.573153 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 17:59:38.705959 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:38.705920 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:39.002545 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:39.002446 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr"] Oct 31 17:59:39.007072 ip-10-0-139-107 kubenswrapper[7618]: W1031 17:59:39.007041 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b5e442_1e92_427f_9bd1_5e60605c3ba5.slice/crio-2ca51bffd5f48f8bbd2760f44c7f2ae8244c23ba0c613774ccdcc0a307077cee WatchSource:0}: Error finding container 2ca51bffd5f48f8bbd2760f44c7f2ae8244c23ba0c613774ccdcc0a307077cee: Status 404 returned error can't find the container with id 2ca51bffd5f48f8bbd2760f44c7f2ae8244c23ba0c613774ccdcc0a307077cee Oct 31 17:59:39.212435 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:39.212392 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" event={"ID":"c1b5e442-1e92-427f-9bd1-5e60605c3ba5","Type":"ContainerStarted","Data":"2ca51bffd5f48f8bbd2760f44c7f2ae8244c23ba0c613774ccdcc0a307077cee"} Oct 31 17:59:40.218612 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:40.218577 7618 generic.go:356] "Generic (PLEG): container finished" podID="c1b5e442-1e92-427f-9bd1-5e60605c3ba5" containerID="d2b9deb858b8ea571628a03c04052f8d50be161ba98bbf1e3115055026bad022" exitCode=0 Oct 31 17:59:40.219041 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:40.218656 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" event={"ID":"c1b5e442-1e92-427f-9bd1-5e60605c3ba5","Type":"ContainerDied","Data":"d2b9deb858b8ea571628a03c04052f8d50be161ba98bbf1e3115055026bad022"} Oct 31 17:59:41.363690 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:41.363663 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:41.481167 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:41.481069 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-openshift-ca\") pod \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\" (UID: \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\") " Oct 31 17:59:41.481386 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:41.481214 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-kube-api-access-dzkn6\" (UniqueName: \"kubernetes.io/projected/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-kube-api-access-dzkn6\") pod \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\" (UID: \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\") " Oct 31 17:59:41.481386 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:41.481324 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-xks-ca\") pod \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\" (UID: \"c1b5e442-1e92-427f-9bd1-5e60605c3ba5\") " Oct 31 17:59:41.481505 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:41.481457 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-openshift-ca" pod "c1b5e442-1e92-427f-9bd1-5e60605c3ba5" (UID: "c1b5e442-1e92-427f-9bd1-5e60605c3ba5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:59:41.481644 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:41.481629 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-openshift-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:59:41.483394 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:41.483374 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-kube-api-access-dzkn6" pod "c1b5e442-1e92-427f-9bd1-5e60605c3ba5" (UID: "c1b5e442-1e92-427f-9bd1-5e60605c3ba5"). InnerVolumeSpecName "kube-api-access-dzkn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:59:41.483508 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:41.483487 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-xks-ca" pod "c1b5e442-1e92-427f-9bd1-5e60605c3ba5" (UID: "c1b5e442-1e92-427f-9bd1-5e60605c3ba5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:59:41.582863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:41.582816 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzkn6\" (UniqueName: \"kubernetes.io/projected/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-kube-api-access-dzkn6\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:59:41.582863 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:41.582857 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b5e442-1e92-427f-9bd1-5e60605c3ba5-maas-api-xks-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 17:59:42.232627 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:42.232589 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" Oct 31 17:59:42.232804 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:42.232598 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5zdr" event={"ID":"c1b5e442-1e92-427f-9bd1-5e60605c3ba5","Type":"ContainerDied","Data":"2ca51bffd5f48f8bbd2760f44c7f2ae8244c23ba0c613774ccdcc0a307077cee"} Oct 31 17:59:42.232804 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:42.232715 7618 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ca51bffd5f48f8bbd2760f44c7f2ae8244c23ba0c613774ccdcc0a307077cee" Oct 31 17:59:49.739545 ip-10-0-139-107 kubenswrapper[7618]: I1031 17:59:49.739504 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-8584d"] Oct 31 18:00:00.141071 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.141036 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw"] Oct 31 18:00:00.141621 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.141536 7618 cpu_manager.go:414] "containerMap: removing container" podUID="c1b5e442-1e92-427f-9bd1-5e60605c3ba5" containerName="revoke-keys" Oct 31 18:00:00.141621 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.141550 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b5e442-1e92-427f-9bd1-5e60605c3ba5" containerName="revoke-keys" Oct 31 18:00:00.141717 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.141653 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b5e442-1e92-427f-9bd1-5e60605c3ba5" containerName="revoke-keys" Oct 31 18:00:00.144685 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.144666 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" Oct 31 18:00:00.159835 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.159806 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 18:00:00.161737 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.161707 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw"] Oct 31 18:00:00.169807 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.169782 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 18:00:00.169938 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.169803 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxlqk\"" Oct 31 18:00:00.287233 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.287198 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a233b53a-9733-449f-8ee1-6bf9eeb3397b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891160-rpbgw\" (UID: \"a233b53a-9733-449f-8ee1-6bf9eeb3397b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" Oct 31 18:00:00.287233 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.287234 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6r5\" (UniqueName: \"kubernetes.io/projected/a233b53a-9733-449f-8ee1-6bf9eeb3397b-kube-api-access-dd6r5\") pod \"maas-api-key-cleanup-29891160-rpbgw\" (UID: \"a233b53a-9733-449f-8ee1-6bf9eeb3397b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" Oct 31 18:00:00.388663 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.388630 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a233b53a-9733-449f-8ee1-6bf9eeb3397b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891160-rpbgw\" (UID: \"a233b53a-9733-449f-8ee1-6bf9eeb3397b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" Oct 31 18:00:00.388663 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.388665 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6r5\" (UniqueName: \"kubernetes.io/projected/a233b53a-9733-449f-8ee1-6bf9eeb3397b-kube-api-access-dd6r5\") pod \"maas-api-key-cleanup-29891160-rpbgw\" (UID: \"a233b53a-9733-449f-8ee1-6bf9eeb3397b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" Oct 31 18:00:00.389342 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.389313 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a233b53a-9733-449f-8ee1-6bf9eeb3397b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891160-rpbgw\" (UID: \"a233b53a-9733-449f-8ee1-6bf9eeb3397b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" Oct 31 18:00:00.405596 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.405535 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6r5\" (UniqueName: \"kubernetes.io/projected/a233b53a-9733-449f-8ee1-6bf9eeb3397b-kube-api-access-dd6r5\") pod \"maas-api-key-cleanup-29891160-rpbgw\" (UID: \"a233b53a-9733-449f-8ee1-6bf9eeb3397b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" Oct 31 18:00:00.457119 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.457078 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" Oct 31 18:00:00.752873 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:00.752800 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw"] Oct 31 18:00:00.756719 ip-10-0-139-107 kubenswrapper[7618]: W1031 18:00:00.756673 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda233b53a_9733_449f_8ee1_6bf9eeb3397b.slice/crio-03aeb8895ba190b43c7ae5097a058b127980caefaa76d4decdedc1594bed5892 WatchSource:0}: Error finding container 03aeb8895ba190b43c7ae5097a058b127980caefaa76d4decdedc1594bed5892: Status 404 returned error can't find the container with id 03aeb8895ba190b43c7ae5097a058b127980caefaa76d4decdedc1594bed5892 Oct 31 18:00:01.348163 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:01.348066 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" event={"ID":"a233b53a-9733-449f-8ee1-6bf9eeb3397b","Type":"ContainerStarted","Data":"88e6ba6b13525971430ad7ada0b3e77c1748fd83ab5aa5a2bd2c2f288bac6277"} Oct 31 18:00:01.348163 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:01.348134 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" event={"ID":"a233b53a-9733-449f-8ee1-6bf9eeb3397b","Type":"ContainerStarted","Data":"03aeb8895ba190b43c7ae5097a058b127980caefaa76d4decdedc1594bed5892"} Oct 31 18:00:01.386036 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:01.385976 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" podStartSLOduration=1.385962857 podStartE2EDuration="1.385962857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:00:01.384575459 +0000 UTC m=+1923.132549927" watchObservedRunningTime="2026-10-31 18:00:01.385962857 +0000 UTC m=+1923.133937380" Oct 31 18:00:02.181195 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:02.181160 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lprq5" Oct 31 18:00:02.355540 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:02.355504 7618 generic.go:356] "Generic (PLEG): container finished" podID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" containerID="88e6ba6b13525971430ad7ada0b3e77c1748fd83ab5aa5a2bd2c2f288bac6277" exitCode=60 Oct 31 18:00:02.355923 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:02.355594 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" event={"ID":"a233b53a-9733-449f-8ee1-6bf9eeb3397b","Type":"ContainerDied","Data":"88e6ba6b13525971430ad7ada0b3e77c1748fd83ab5aa5a2bd2c2f288bac6277"} Oct 31 18:00:02.355923 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:02.355915 7618 scope.go:122] "RemoveContainer" containerID="88e6ba6b13525971430ad7ada0b3e77c1748fd83ab5aa5a2bd2c2f288bac6277" Oct 31 18:00:03.362913 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:03.362876 7618 generic.go:356] "Generic (PLEG): container finished" podID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" containerID="aa1de02e6a135357b9588ba88458f77d2a028089cdb4169a83f9d5fbd9f04ba7" exitCode=60 Oct 31 18:00:03.363458 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:03.362960 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" event={"ID":"a233b53a-9733-449f-8ee1-6bf9eeb3397b","Type":"ContainerDied","Data":"aa1de02e6a135357b9588ba88458f77d2a028089cdb4169a83f9d5fbd9f04ba7"} Oct 31 18:00:03.363458 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:03.363045 7618 scope.go:122] "RemoveContainer" containerID="88e6ba6b13525971430ad7ada0b3e77c1748fd83ab5aa5a2bd2c2f288bac6277" Oct 31 18:00:03.363458 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:03.363350 7618 scope.go:122] "RemoveContainer" containerID="aa1de02e6a135357b9588ba88458f77d2a028089cdb4169a83f9d5fbd9f04ba7" Oct 31 18:00:03.363661 ip-10-0-139-107 kubenswrapper[7618]: E1031 18:00:03.363634 7618 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891160-rpbgw_odh-ai-gateway-infra(a233b53a-9733-449f-8ee1-6bf9eeb3397b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" Oct 31 18:00:04.371320 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:04.371285 7618 scope.go:122] "RemoveContainer" containerID="aa1de02e6a135357b9588ba88458f77d2a028089cdb4169a83f9d5fbd9f04ba7" Oct 31 18:00:04.371747 ip-10-0-139-107 kubenswrapper[7618]: E1031 18:00:04.371590 7618 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891160-rpbgw_odh-ai-gateway-infra(a233b53a-9733-449f-8ee1-6bf9eeb3397b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" Oct 31 18:00:08.572359 ip-10-0-139-107 kubenswrapper[7618]: E1031 18:00:08.572321 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 18:00:15.835646 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:15.835615 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lprq5_28290dd1-c75b-4c13-88ab-2f9a6dea4795/manager/1.log" Oct 31 18:00:16.233657 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:16.233622 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2t6x8_0672b07e-809a-443d-9dd8-6df70e1e5ddf/manager/1.log" Oct 31 18:00:16.701961 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:16.701935 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6bncn_4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d/server/0.log" Oct 31 18:00:16.897935 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:16.897896 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-r6mcs_e9da2c45-46f7-4b1a-8607-069282d70b2d/manager/0.log" Oct 31 18:00:17.127813 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:17.127740 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5955s_e77051af-e037-4465-8718-3c8322187994/proxy/0.log" Oct 31 18:00:17.366543 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:17.366510 7618 scope.go:122] "RemoveContainer" containerID="aa1de02e6a135357b9588ba88458f77d2a028089cdb4169a83f9d5fbd9f04ba7" Oct 31 18:00:17.531210 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:17.531181 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lprq5_28290dd1-c75b-4c13-88ab-2f9a6dea4795/manager/0.log" Oct 31 18:00:17.919525 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:17.919480 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2t6x8_0672b07e-809a-443d-9dd8-6df70e1e5ddf/manager/0.log" Oct 31 18:00:18.444821 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:18.444788 7618 generic.go:356] "Generic (PLEG): container finished" podID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" containerID="daddaf38850fb8f85867addc0ff9d01e732be28dd12811ca9d11565b3367c8bb" exitCode=60 Oct 31 18:00:18.444982 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:18.444857 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" event={"ID":"a233b53a-9733-449f-8ee1-6bf9eeb3397b","Type":"ContainerDied","Data":"daddaf38850fb8f85867addc0ff9d01e732be28dd12811ca9d11565b3367c8bb"} Oct 31 18:00:18.444982 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:18.444913 7618 scope.go:122] "RemoveContainer" containerID="aa1de02e6a135357b9588ba88458f77d2a028089cdb4169a83f9d5fbd9f04ba7" Oct 31 18:00:18.445338 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:18.445318 7618 scope.go:122] "RemoveContainer" containerID="daddaf38850fb8f85867addc0ff9d01e732be28dd12811ca9d11565b3367c8bb" Oct 31 18:00:18.445588 ip-10-0-139-107 kubenswrapper[7618]: E1031 18:00:18.445568 7618 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891160-rpbgw_odh-ai-gateway-infra(a233b53a-9733-449f-8ee1-6bf9eeb3397b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" Oct 31 18:00:18.841834 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:18.841801 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29891160-rpbgw_a233b53a-9733-449f-8ee1-6bf9eeb3397b/cleanup/2.log" Oct 31 18:00:19.018961 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:19.018929 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-6fg2r_630e7556-9a56-4ee8-a565-0dd5167ea24a/revoke-keys/0.log" Oct 31 18:00:19.487137 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:19.487104 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw"] Oct 31 18:00:19.565328 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:19.565218 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-s5zdr_c1b5e442-1e92-427f-9bd1-5e60605c3ba5/revoke-keys/0.log" Oct 31 18:00:19.630763 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:19.630735 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" Oct 31 18:00:19.674916 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:19.674876 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a233b53a-9733-449f-8ee1-6bf9eeb3397b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a233b53a-9733-449f-8ee1-6bf9eeb3397b-maas-api-service-ca\") pod \"a233b53a-9733-449f-8ee1-6bf9eeb3397b\" (UID: \"a233b53a-9733-449f-8ee1-6bf9eeb3397b\") " Oct 31 18:00:19.675101 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:19.674968 7618 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a233b53a-9733-449f-8ee1-6bf9eeb3397b-kube-api-access-dd6r5\" (UniqueName: \"kubernetes.io/projected/a233b53a-9733-449f-8ee1-6bf9eeb3397b-kube-api-access-dd6r5\") pod \"a233b53a-9733-449f-8ee1-6bf9eeb3397b\" (UID: \"a233b53a-9733-449f-8ee1-6bf9eeb3397b\") " Oct 31 18:00:19.675430 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:19.675287 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a233b53a-9733-449f-8ee1-6bf9eeb3397b-maas-api-service-ca" pod "a233b53a-9733-449f-8ee1-6bf9eeb3397b" (UID: "a233b53a-9733-449f-8ee1-6bf9eeb3397b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 18:00:19.675430 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:19.675408 7618 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a233b53a-9733-449f-8ee1-6bf9eeb3397b-maas-api-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 18:00:19.677976 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:19.677951 7618 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a233b53a-9733-449f-8ee1-6bf9eeb3397b-kube-api-access-dd6r5" pod "a233b53a-9733-449f-8ee1-6bf9eeb3397b" (UID: "a233b53a-9733-449f-8ee1-6bf9eeb3397b"). InnerVolumeSpecName "kube-api-access-dd6r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 18:00:19.746585 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:19.746504 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-gbnh8_df61d6f8-c549-4e80-b4bb-18d2e11f8fc0/revoke-keys/0.log" Oct 31 18:00:19.776840 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:19.776807 7618 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd6r5\" (UniqueName: \"kubernetes.io/projected/a233b53a-9733-449f-8ee1-6bf9eeb3397b-kube-api-access-dd6r5\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Oct 31 18:00:20.457725 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:20.457692 7618 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" Oct 31 18:00:20.458190 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:20.457686 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw" event={"ID":"a233b53a-9733-449f-8ee1-6bf9eeb3397b","Type":"ContainerDied","Data":"03aeb8895ba190b43c7ae5097a058b127980caefaa76d4decdedc1594bed5892"} Oct 31 18:00:20.458190 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:20.457824 7618 scope.go:122] "RemoveContainer" containerID="daddaf38850fb8f85867addc0ff9d01e732be28dd12811ca9d11565b3367c8bb" Oct 31 18:00:20.504562 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:20.504526 7618 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw"] Oct 31 18:00:20.514748 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:20.514715 7618 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891160-rpbgw"] Oct 31 18:00:22.217643 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:22.217616 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-548jz_83290f73-1c39-48d4-a226-490f6dc869fc/kuadrant-console-plugin/0.log" Oct 31 18:00:22.372213 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:22.372177 7618 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" path="/var/lib/kubelet/pods/a233b53a-9733-449f-8ee1-6bf9eeb3397b/volumes" Oct 31 18:00:22.407896 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:22.407862 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5jgkf_893148fa-6f42-4363-ada7-0108932094f2/manager/1.log" Oct 31 18:00:22.592870 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:22.592792 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8584d_165b216f-bc57-419e-ad86-fe80b6d25168/limitador/0.log" Oct 31 18:00:23.198746 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:23.198709 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5jgkf_893148fa-6f42-4363-ada7-0108932094f2/manager/0.log" Oct 31 18:00:24.404811 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:24.404787 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77485b4dcc-56r9l_ff97ed13-67d4-4416-8246-5b99310096b0/kube-auth-proxy/0.log" Oct 31 18:00:24.778093 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:24.778018 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-4vwr9_e63bb320-851c-4bd6-880b-906a3c12bbda/istio-proxy/0.log" Oct 31 18:00:25.680140 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:25.680112 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-hk9mq_f04cec43-eec4-4872-8f9b-8609fc89a551/istio-proxy/0.log" Oct 31 18:00:25.854679 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:25.854644 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-kd6mw_df608734-a9f8-40c6-9430-1b25fa3389e4/istio-proxy/0.log" Oct 31 18:00:38.572202 ip-10-0-139-107 kubenswrapper[7618]: E1031 18:00:38.572163 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 18:00:39.186581 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:39.186549 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lprq5_28290dd1-c75b-4c13-88ab-2f9a6dea4795/manager/1.log" Oct 31 18:00:39.587019 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:39.586936 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2t6x8_0672b07e-809a-443d-9dd8-6df70e1e5ddf/manager/1.log" Oct 31 18:00:40.038044 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:40.038014 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6bncn_4de841b8-0cdf-4cab-ac1f-fe2df4cdbd9d/server/0.log" Oct 31 18:00:40.237655 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:40.237622 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-r6mcs_e9da2c45-46f7-4b1a-8607-069282d70b2d/manager/0.log" Oct 31 18:00:40.459989 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:40.459955 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5955s_e77051af-e037-4465-8718-3c8322187994/proxy/0.log" Oct 31 18:00:41.372615 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:41.372585 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-6fg2r_630e7556-9a56-4ee8-a565-0dd5167ea24a/revoke-keys/0.log" Oct 31 18:00:41.728874 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:41.728847 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-s5zdr_c1b5e442-1e92-427f-9bd1-5e60605c3ba5/revoke-keys/0.log" Oct 31 18:00:41.910553 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:41.910525 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-gbnh8_df61d6f8-c549-4e80-b4bb-18d2e11f8fc0/revoke-keys/0.log" Oct 31 18:00:43.774323 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:43.774294 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-548jz_83290f73-1c39-48d4-a226-490f6dc869fc/kuadrant-console-plugin/0.log" Oct 31 18:00:43.962540 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:43.962507 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5jgkf_893148fa-6f42-4363-ada7-0108932094f2/manager/1.log" Oct 31 18:00:44.160570 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:44.160543 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8584d_165b216f-bc57-419e-ad86-fe80b6d25168/limitador/0.log" Oct 31 18:00:45.300942 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:45.300913 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77485b4dcc-56r9l_ff97ed13-67d4-4416-8246-5b99310096b0/kube-auth-proxy/0.log" Oct 31 18:00:45.683351 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:45.683319 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-4vwr9_e63bb320-851c-4bd6-880b-906a3c12bbda/istio-proxy/0.log" Oct 31 18:00:46.634781 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:46.634751 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-hk9mq_f04cec43-eec4-4872-8f9b-8609fc89a551/istio-proxy/0.log" Oct 31 18:00:46.826294 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:00:46.826262 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-kd6mw_df608734-a9f8-40c6-9430-1b25fa3389e4/istio-proxy/0.log" Oct 31 18:01:08.571327 ip-10-0-139-107 kubenswrapper[7618]: E1031 18:01:08.571288 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 18:01:38.571275 ip-10-0-139-107 kubenswrapper[7618]: E1031 18:01:38.571181 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 18:02:08.571812 ip-10-0-139-107 kubenswrapper[7618]: E1031 18:02:08.571707 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 18:02:14.666144 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:14.666110 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-swbqv_bec5b1da-4675-4713-9e51-9e7be00b5ba0/global-pull-secret-syncer/0.log" Oct 31 18:02:15.366433 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:15.366399 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v9fs8_76940cd0-ef63-4e10-965e-e871ab476281/konnectivity-agent/0.log" Oct 31 18:02:15.864796 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:15.864764 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-107.ec2.internal_8389bef8c4009e269aafe36de827cd22/haproxy/0.log" Oct 31 18:02:22.574982 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:22.574957 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-548jz_83290f73-1c39-48d4-a226-490f6dc869fc/kuadrant-console-plugin/0.log" Oct 31 18:02:22.706275 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:22.705791 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5jgkf_893148fa-6f42-4363-ada7-0108932094f2/manager/1.log" Oct 31 18:02:22.729193 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:22.729158 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5jgkf_893148fa-6f42-4363-ada7-0108932094f2/manager/0.log" Oct 31 18:02:22.839696 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:22.839617 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-8584d_165b216f-bc57-419e-ad86-fe80b6d25168/limitador/0.log" Oct 31 18:02:24.353950 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.353913 7618 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-csf86/perf-node-gather-daemonset-l7745"] Oct 31 18:02:24.354526 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.354505 7618 cpu_manager.go:414] "containerMap: removing container" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" containerName="cleanup" Oct 31 18:02:24.354575 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.354528 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" containerName="cleanup" Oct 31 18:02:24.354575 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.354548 7618 cpu_manager.go:414] "containerMap: removing container" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" containerName="cleanup" Oct 31 18:02:24.354575 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.354553 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" containerName="cleanup" Oct 31 18:02:24.354575 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.354561 7618 cpu_manager.go:414] "containerMap: removing container" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" containerName="cleanup" Oct 31 18:02:24.354575 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.354566 7618 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" containerName="cleanup" Oct 31 18:02:24.354747 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.354654 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" containerName="cleanup" Oct 31 18:02:24.354747 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.354670 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" containerName="cleanup" Oct 31 18:02:24.354747 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.354678 7618 memory_manager.go:365] "RemoveStaleState removing state" podUID="a233b53a-9733-449f-8ee1-6bf9eeb3397b" containerName="cleanup" Oct 31 18:02:24.357704 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.357689 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.372617 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.372581 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-csf86\"/\"openshift-service-ca.crt\"" Oct 31 18:02:24.374312 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.374285 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-csf86/perf-node-gather-daemonset-l7745"] Oct 31 18:02:24.382758 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.382732 7618 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-csf86\"/\"kube-root-ca.crt\"" Oct 31 18:02:24.382933 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.382913 7618 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-csf86\"/\"default-dockercfg-5qfvc\"" Oct 31 18:02:24.436042 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.435997 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwnx4\" (UniqueName: \"kubernetes.io/projected/07b5f4d6-550e-443c-b652-7662884dc587-kube-api-access-bwnx4\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.436225 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.436054 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07b5f4d6-550e-443c-b652-7662884dc587-sys\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.436225 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.436184 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07b5f4d6-550e-443c-b652-7662884dc587-lib-modules\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.436333 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.436310 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07b5f4d6-550e-443c-b652-7662884dc587-proc\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.436421 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.436406 7618 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07b5f4d6-550e-443c-b652-7662884dc587-podres\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.537566 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.537526 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwnx4\" (UniqueName: \"kubernetes.io/projected/07b5f4d6-550e-443c-b652-7662884dc587-kube-api-access-bwnx4\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.537776 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.537573 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07b5f4d6-550e-443c-b652-7662884dc587-sys\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.537776 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.537621 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07b5f4d6-550e-443c-b652-7662884dc587-lib-modules\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.537776 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.537676 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07b5f4d6-550e-443c-b652-7662884dc587-sys\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.537776 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.537680 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07b5f4d6-550e-443c-b652-7662884dc587-proc\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.537776 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.537721 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07b5f4d6-550e-443c-b652-7662884dc587-proc\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.538044 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.537791 7618 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07b5f4d6-550e-443c-b652-7662884dc587-podres\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.538044 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.537793 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07b5f4d6-550e-443c-b652-7662884dc587-lib-modules\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.538044 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.537892 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07b5f4d6-550e-443c-b652-7662884dc587-podres\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.538241 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.538224 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03ebb483-bb1a-471c-9363-0a8674cec02d/alertmanager/0.log" Oct 31 18:02:24.554268 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.554226 7618 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwnx4\" (UniqueName: \"kubernetes.io/projected/07b5f4d6-550e-443c-b652-7662884dc587-kube-api-access-bwnx4\") pod \"perf-node-gather-daemonset-l7745\" (UID: \"07b5f4d6-550e-443c-b652-7662884dc587\") " pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.656587 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.656549 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03ebb483-bb1a-471c-9363-0a8674cec02d/config-reloader/0.log" Oct 31 18:02:24.670574 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.670542 7618 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:24.774768 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.774741 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03ebb483-bb1a-471c-9363-0a8674cec02d/kube-rbac-proxy-web/0.log" Oct 31 18:02:24.892631 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.892601 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03ebb483-bb1a-471c-9363-0a8674cec02d/kube-rbac-proxy/0.log" Oct 31 18:02:24.963647 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:24.963569 7618 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-csf86/perf-node-gather-daemonset-l7745"] Oct 31 18:02:24.966902 ip-10-0-139-107 kubenswrapper[7618]: W1031 18:02:24.966875 7618 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07b5f4d6_550e_443c_b652_7662884dc587.slice/crio-09f2e87328d2960644eb91c6d65379a2c0b39626353e7b859182ed9988bfb743 WatchSource:0}: Error finding container 09f2e87328d2960644eb91c6d65379a2c0b39626353e7b859182ed9988bfb743: Status 404 returned error can't find the container with id 09f2e87328d2960644eb91c6d65379a2c0b39626353e7b859182ed9988bfb743 Oct 31 18:02:25.010203 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:25.010173 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03ebb483-bb1a-471c-9363-0a8674cec02d/kube-rbac-proxy-metric/0.log" Oct 31 18:02:25.100026 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:25.099989 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" event={"ID":"07b5f4d6-550e-443c-b652-7662884dc587","Type":"ContainerStarted","Data":"c84e59bf0f7610edc803d4fc4be0c3ea9618d0e1102938644c4b3c598916f265"} Oct 31 18:02:25.100238 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:25.100046 7618 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" event={"ID":"07b5f4d6-550e-443c-b652-7662884dc587","Type":"ContainerStarted","Data":"09f2e87328d2960644eb91c6d65379a2c0b39626353e7b859182ed9988bfb743"} Oct 31 18:02:25.100238 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:25.100147 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:25.127396 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:25.127369 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03ebb483-bb1a-471c-9363-0a8674cec02d/prom-label-proxy/0.log" Oct 31 18:02:25.136588 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:25.136542 7618 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" podStartSLOduration=1.136529581 podStartE2EDuration="1.136529581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:02:25.134463658 +0000 UTC m=+2066.882438127" watchObservedRunningTime="2026-10-31 18:02:25.136529581 +0000 UTC m=+2066.884504047" Oct 31 18:02:25.246062 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:25.245993 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03ebb483-bb1a-471c-9363-0a8674cec02d/init-config-reloader/0.log" Oct 31 18:02:25.497301 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:25.497189 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-spjzf_9206f5fc-3e4d-4574-8c55-aaca1a33e3d4/kube-state-metrics/0.log" Oct 31 18:02:25.612016 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:25.611987 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-spjzf_9206f5fc-3e4d-4574-8c55-aaca1a33e3d4/kube-rbac-proxy-main/0.log" Oct 31 18:02:25.731673 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:25.731646 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-spjzf_9206f5fc-3e4d-4574-8c55-aaca1a33e3d4/kube-rbac-proxy-self/0.log" Oct 31 18:02:25.852555 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:25.852483 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5db8c4796b-lwk5p_58a5dd3d-3ace-4a01-b358-d418d1fa81aa/metrics-server/0.log" Oct 31 18:02:25.971803 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:25.971773 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-t2v2v_bd2a7b5c-fbe5-43d3-9121-403946f33aca/monitoring-plugin/0.log" Oct 31 18:02:26.095415 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:26.095385 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66bdk_93b4b2af-0bd0-4907-9535-0ce4ef46383c/node-exporter/0.log" Oct 31 18:02:26.213582 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:26.213555 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66bdk_93b4b2af-0bd0-4907-9535-0ce4ef46383c/kube-rbac-proxy/0.log" Oct 31 18:02:26.330882 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:26.330853 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66bdk_93b4b2af-0bd0-4907-9535-0ce4ef46383c/init-textfile/0.log" Oct 31 18:02:27.165341 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:27.165312 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-drfrv_6e895668-db69-4458-9e3d-fcd82e31084b/kube-rbac-proxy-main/0.log" Oct 31 18:02:27.282791 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:27.282757 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-drfrv_6e895668-db69-4458-9e3d-fcd82e31084b/kube-rbac-proxy-self/0.log" Oct 31 18:02:27.401342 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:27.401308 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-drfrv_6e895668-db69-4458-9e3d-fcd82e31084b/openshift-state-metrics/0.log" Oct 31 18:02:27.529338 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:27.529238 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3caff88e-0edc-4fac-9a1c-1387610e526f/prometheus/0.log" Oct 31 18:02:27.647556 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:27.647520 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3caff88e-0edc-4fac-9a1c-1387610e526f/config-reloader/0.log" Oct 31 18:02:27.766399 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:27.766368 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3caff88e-0edc-4fac-9a1c-1387610e526f/thanos-sidecar/0.log" Oct 31 18:02:27.883994 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:27.883921 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3caff88e-0edc-4fac-9a1c-1387610e526f/kube-rbac-proxy-web/0.log" Oct 31 18:02:28.002155 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:28.002127 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3caff88e-0edc-4fac-9a1c-1387610e526f/kube-rbac-proxy/0.log" Oct 31 18:02:28.120288 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:28.120240 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3caff88e-0edc-4fac-9a1c-1387610e526f/kube-rbac-proxy-thanos/0.log" Oct 31 18:02:28.242965 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:28.242940 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3caff88e-0edc-4fac-9a1c-1387610e526f/init-config-reloader/0.log" Oct 31 18:02:28.729313 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:28.729276 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f6f8d64d6-sc4qk_2269cc9c-7732-4144-bc2e-8dafebdd1957/telemeter-client/0.log" Oct 31 18:02:28.849047 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:28.849010 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f6f8d64d6-sc4qk_2269cc9c-7732-4144-bc2e-8dafebdd1957/reload/0.log" Oct 31 18:02:28.966867 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:28.966829 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f6f8d64d6-sc4qk_2269cc9c-7732-4144-bc2e-8dafebdd1957/kube-rbac-proxy/0.log" Oct 31 18:02:29.095820 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:29.095743 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf7478d4d-qqjlw_016aec04-e1a2-4dd4-8a59-9cfc0418da6f/thanos-query/0.log" Oct 31 18:02:29.214398 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:29.214371 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf7478d4d-qqjlw_016aec04-e1a2-4dd4-8a59-9cfc0418da6f/kube-rbac-proxy-web/0.log" Oct 31 18:02:29.331568 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:29.331539 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf7478d4d-qqjlw_016aec04-e1a2-4dd4-8a59-9cfc0418da6f/kube-rbac-proxy/0.log" Oct 31 18:02:29.450097 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:29.450067 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf7478d4d-qqjlw_016aec04-e1a2-4dd4-8a59-9cfc0418da6f/prom-label-proxy/0.log" Oct 31 18:02:29.567897 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:29.567864 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf7478d4d-qqjlw_016aec04-e1a2-4dd4-8a59-9cfc0418da6f/kube-rbac-proxy-rules/0.log" Oct 31 18:02:29.686473 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:29.686444 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf7478d4d-qqjlw_016aec04-e1a2-4dd4-8a59-9cfc0418da6f/kube-rbac-proxy-metrics/0.log" Oct 31 18:02:31.115909 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:31.115880 7618 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-csf86/perf-node-gather-daemonset-l7745" Oct 31 18:02:36.760649 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:36.760617 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4jjks_bd29fa4f-f33c-4c02-988c-afcfb47b6ca0/dns/0.log" Oct 31 18:02:36.883750 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:36.883719 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4jjks_bd29fa4f-f33c-4c02-988c-afcfb47b6ca0/kube-rbac-proxy/0.log" Oct 31 18:02:37.490196 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:37.490163 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-58mtk_159b2703-3b66-4ddc-8762-b4546b942282/dns-node-resolver/0.log" Oct 31 18:02:37.897311 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:37.897280 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77485b4dcc-56r9l_ff97ed13-67d4-4416-8246-5b99310096b0/kube-auth-proxy/0.log" Oct 31 18:02:38.027602 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:38.027576 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-4vwr9_e63bb320-851c-4bd6-880b-906a3c12bbda/istio-proxy/0.log" Oct 31 18:02:38.335707 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:38.335640 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-hk9mq_f04cec43-eec4-4872-8f9b-8609fc89a551/istio-proxy/0.log" Oct 31 18:02:38.401328 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:38.401296 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-kd6mw_df608734-a9f8-40c6-9430-1b25fa3389e4/istio-proxy/0.log" Oct 31 18:02:38.572400 ip-10-0-139-107 kubenswrapper[7618]: E1031 18:02:38.572364 7618 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-knkx9" containerName="extractor" Oct 31 18:02:38.881181 ip-10-0-139-107 kubenswrapper[7618]: I1031 18:02:38.881156 7618 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9789n_806f6a75-275f-48d2-ae95-03e1a5e46515/node-ca/0.log"