Nov 02 19:49:13.403324 ip-10-0-134-194 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 19:49:13.403335 ip-10-0-134-194 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 19:49:13.403342 ip-10-0-134-194 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 19:49:13.403562 ip-10-0-134-194 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 19:49:23.576333 ip-10-0-134-194 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 19:49:23.576351 ip-10-0-134-194 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot bd7e8f4ab11a4480b779d12b34fa6dd1 -- Nov 02 19:51:30.659165 ip-10-0-134-194 systemd[1]: Starting Kubernetes Kubelet... Nov 02 19:51:31.039360 ip-10-0-134-194 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:51:31.039360 ip-10-0-134-194 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 19:51:31.039360 ip-10-0-134-194 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:51:31.039360 ip-10-0-134-194 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:51:31.044076 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.043990 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:31.044076 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044076 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:31.044076 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044080 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044083 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044087 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044090 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044093 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044096 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044099 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044101 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044103 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044106 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044109 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044114 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044118 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044121 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044124 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044127 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044130 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044141 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044144 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044146 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:31.044165 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044149 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044152 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044154 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044157 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044159 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044162 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044164 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044167 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044171 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044173 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044176 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044179 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044182 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044185 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044187 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044190 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044193 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044195 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044198 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:31.044629 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044202 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044206 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044208 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044211 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044213 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044216 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044218 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044221 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044223 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044226 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044228 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044232 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044235 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044238 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044242 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044245 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044247 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044250 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044253 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044256 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:31.045098 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044258 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044261 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044263 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044266 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044268 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044271 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044273 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044276 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044278 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044281 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044283 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044286 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044289 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044291 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044293 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044296 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044299 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044301 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044303 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044306 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:31.045574 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044309 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044311 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044314 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044316 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044320 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044323 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044326 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044329 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044332 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044334 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044337 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044340 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044342 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044344 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044347 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044350 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044352 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044355 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044358 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:31.046056 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044361 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044363 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044366 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044368 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044371 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.044373 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045170 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045178 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045181 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045184 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045186 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045189 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045192 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045194 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045198 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045200 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045203 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045206 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045209 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:31.046499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045212 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045216 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045219 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045222 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045224 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045227 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045230 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045233 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045236 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045238 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045241 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045244 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045247 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045249 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045252 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045255 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045258 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045260 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045263 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:31.047007 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045266 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045268 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045271 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045273 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045276 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045279 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045281 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045283 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045286 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045288 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045291 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045294 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045297 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045300 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045304 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045308 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045312 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045315 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045318 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:31.047464 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045321 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045323 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045326 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045328 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045331 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045333 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045336 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045338 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045341 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045343 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045346 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045348 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045351 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045353 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045356 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045358 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045362 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045364 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045367 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045370 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:31.048026 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045372 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045375 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045377 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045381 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045384 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045386 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045389 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045393 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045395 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045400 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045404 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045407 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045410 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045413 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045415 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045418 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045421 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045423 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045426 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:31.048490 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045428 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045431 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045434 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045436 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045438 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045441 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045444 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045446 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045448 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045451 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045453 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045457 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045460 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045463 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045465 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045468 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045949 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045955 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045959 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:31.048963 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045962 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045965 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045968 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045970 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045973 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045976 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045978 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045981 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045984 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045987 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045989 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045992 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045995 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.045997 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046000 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046003 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046006 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046009 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046011 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046014 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:31.049419 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046016 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046019 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046022 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046024 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046027 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046030 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046035 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046038 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046040 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046043 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046046 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046048 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046051 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046054 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046057 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046060 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046063 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046066 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046069 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046071 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:31.049894 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046074 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046076 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046079 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046081 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046084 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046087 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046089 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046092 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046094 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046097 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046100 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046103 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046105 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046108 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046110 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046114 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046116 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046120 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046125 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:31.050360 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046129 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046133 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046136 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046140 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046144 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046147 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046150 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046153 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046156 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046159 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046163 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046166 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046168 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046171 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046173 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046176 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046179 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046181 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046184 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:31.050810 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046186 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046190 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046193 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046195 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046198 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046201 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046203 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046206 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046208 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046211 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046214 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046216 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046220 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046222 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046225 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046229 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046231 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046234 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046236 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:31.051268 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046239 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046241 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046244 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046247 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046249 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.046252 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047279 2575 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047289 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047295 2575 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047301 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047305 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047309 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047313 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047317 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047322 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047326 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047329 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047333 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047336 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047339 2575 flags.go:64] FLAG: --cgroup-root="" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047342 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047345 2575 flags.go:64] FLAG: --client-ca-file="" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047348 2575 flags.go:64] FLAG: --cloud-provider="external" Nov 02 19:51:31.051745 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047351 2575 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047357 2575 flags.go:64] FLAG: --cluster-domain="" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047360 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047363 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047366 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047370 2575 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047374 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047378 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047381 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047384 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047387 2575 flags.go:64] FLAG: --contention-profiling="false" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047390 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047393 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047397 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047400 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047404 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047407 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047410 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047413 2575 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047416 2575 flags.go:64] FLAG: --enable-server="true" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047419 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047424 2575 flags.go:64] FLAG: --event-burst="100" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047427 2575 flags.go:64] FLAG: --event-qps="50" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047431 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 19:51:31.052282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047435 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047438 2575 flags.go:64] FLAG: --eviction-hard="" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047442 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047446 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047449 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047452 2575 flags.go:64] FLAG: --eviction-soft="" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047455 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047458 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047461 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047464 2575 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047467 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047470 2575 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047473 2575 flags.go:64] FLAG: --feature-gates="" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047478 2575 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047481 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047485 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047488 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047491 2575 flags.go:64] FLAG: --healthz-port="10248" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047494 2575 flags.go:64] FLAG: --help="false" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047497 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047501 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047504 2575 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047507 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 19:51:31.052862 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047511 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047514 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047517 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047520 2575 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047523 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047526 2575 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047529 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047533 2575 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047538 2575 flags.go:64] FLAG: --kube-reserved="" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047541 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047544 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047547 2575 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047550 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047553 2575 flags.go:64] FLAG: --lock-file="" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047556 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047559 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047562 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047568 2575 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047571 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047575 2575 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047577 2575 flags.go:64] FLAG: --logging-format="text" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047580 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047584 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047587 2575 flags.go:64] FLAG: --manifest-url="" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047590 2575 flags.go:64] FLAG: --manifest-url-header="" Nov 02 19:51:31.053467 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047594 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047597 2575 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047601 2575 flags.go:64] FLAG: --max-pods="110" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047604 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047607 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047610 2575 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047613 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047621 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047625 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047628 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047635 2575 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047639 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047642 2575 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047645 2575 flags.go:64] FLAG: --pod-cidr="" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047648 2575 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047652 2575 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047655 2575 flags.go:64] FLAG: --pods-per-core="0" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047658 2575 flags.go:64] FLAG: --port="10250" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047661 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047664 2575 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-012c7be788c730f37" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047668 2575 flags.go:64] FLAG: --qos-reserved="" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047671 2575 flags.go:64] FLAG: --read-only-port="10255" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047674 2575 flags.go:64] FLAG: --register-node="true" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047677 2575 flags.go:64] FLAG: --register-with-taints="" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047681 2575 flags.go:64] FLAG: --registry-burst="10" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047684 2575 flags.go:64] FLAG: --registry-qps="5" Nov 02 19:51:31.054064 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047687 2575 flags.go:64] FLAG: --reserved-cpus="" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047690 2575 flags.go:64] FLAG: --reserved-memory="" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047694 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047697 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047701 2575 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047703 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047706 2575 flags.go:64] FLAG: --runonce="false" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047709 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047713 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047716 2575 flags.go:64] FLAG: --seccomp-default="false" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047719 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047722 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047725 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047728 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047731 2575 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047734 2575 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047737 2575 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047740 2575 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047743 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047746 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047749 2575 flags.go:64] FLAG: --system-cgroups="" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047753 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047759 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047762 2575 flags.go:64] FLAG: --tls-cert-file="" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047765 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 19:51:31.054683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047769 2575 flags.go:64] FLAG: --tls-min-version="" Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047772 2575 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047775 2575 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047778 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047783 2575 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047786 2575 flags.go:64] FLAG: --v="2" Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047791 2575 flags.go:64] FLAG: --version="false" Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047795 2575 flags.go:64] FLAG: --vmodule="" Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047799 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.047803 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047951 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047955 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047959 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047961 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047964 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047966 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047969 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047972 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047975 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047977 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047980 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047982 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:31.055499 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047988 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047991 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047994 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047996 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.047999 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048001 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048006 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048008 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048011 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048014 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048016 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048019 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048021 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048024 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048028 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048031 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048034 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048036 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048039 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048042 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:31.056141 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048045 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048047 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048050 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048053 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048055 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048058 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048060 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048063 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048065 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048068 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048071 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048073 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048076 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048080 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048083 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048085 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048088 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048091 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048094 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:31.056665 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048097 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048099 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048102 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048105 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048107 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048110 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048112 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048116 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048119 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048122 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048124 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048127 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048130 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048132 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048135 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048138 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048141 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048143 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048146 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:31.057306 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048148 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048151 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048153 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048156 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048159 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048161 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048164 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048167 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048170 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048172 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048175 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048178 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048180 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048183 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048185 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048188 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048190 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048193 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048196 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:31.057757 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048200 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048205 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048209 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048212 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048215 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048218 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048221 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048224 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048228 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048231 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048234 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048237 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048239 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048242 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048244 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048247 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:31.058221 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048250 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.048277 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048317 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048322 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048325 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048327 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048330 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048333 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048336 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048339 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048341 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048344 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048347 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048350 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048352 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048355 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:31.058594 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048357 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048360 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048363 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048366 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048368 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048371 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048374 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048376 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048379 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048382 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048384 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048387 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048389 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048392 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048394 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048397 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048399 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048402 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048405 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048408 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:31.058991 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048410 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048413 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048415 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048418 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048421 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048424 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048426 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048429 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048432 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048435 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048438 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048440 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048443 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048445 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048449 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048452 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048454 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048457 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048459 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:31.059456 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048462 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048465 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048467 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048470 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048472 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048475 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048478 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048481 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048483 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048485 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048489 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048492 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048495 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048498 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048501 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048503 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048507 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048510 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048513 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:31.059915 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048516 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048520 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048524 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048526 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048529 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048532 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048534 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048538 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048543 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048547 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048549 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048552 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048554 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048557 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048559 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048562 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048565 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048567 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048570 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:31.060364 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048573 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048575 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048578 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048580 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048584 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048587 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048590 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048592 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048595 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048597 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048600 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048603 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048606 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048609 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.048611 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.055986 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.056003 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.056761 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.056802 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057248 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:31.060850 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057259 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057264 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057267 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057270 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057273 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057276 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057278 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057281 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057284 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057287 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057289 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057292 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057294 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057297 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057300 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057303 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057306 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057309 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057312 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:31.061380 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057315 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057318 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057321 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057324 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057327 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057330 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057334 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057337 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057340 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057342 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057345 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057353 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057356 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057359 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057362 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057364 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057367 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057370 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057372 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057375 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:31.061827 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057377 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057382 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057386 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057389 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057392 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057394 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057397 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057400 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057404 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057406 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057410 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057412 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057415 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057418 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057420 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057423 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057426 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057429 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057432 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:31.062311 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057435 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057438 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057441 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057443 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057446 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057448 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057451 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057454 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057457 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057459 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057463 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057467 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057470 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057473 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057476 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057478 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057481 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057483 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057486 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057488 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:31.062762 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057491 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057494 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057496 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057499 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057502 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057504 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057507 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057509 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057512 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057515 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057518 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057520 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057523 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057525 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057528 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057531 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057533 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057536 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057539 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057541 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:31.063241 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057544 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057546 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057549 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057551 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057554 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057556 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057559 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.057588 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057754 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057760 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057763 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057766 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057769 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057778 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057781 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057783 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:31.063717 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057786 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057789 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057792 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057794 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057797 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057800 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057802 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057805 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057807 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057810 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057812 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057815 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057818 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057820 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057823 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057825 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057828 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057831 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057833 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:31.064117 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057836 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057838 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057841 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057844 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057846 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057849 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057851 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057853 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057856 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057859 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057861 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057870 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057887 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057890 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057893 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057895 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057898 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057901 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057903 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057906 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:31.064567 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057909 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057911 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057914 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057916 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057920 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057922 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057925 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057928 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057932 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057936 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057940 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057943 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057946 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057948 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057951 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057953 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057956 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057958 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:31.065110 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057961 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057964 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057966 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057969 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057972 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057974 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057982 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057985 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057987 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057990 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057992 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057995 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.057997 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058000 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058003 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058006 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058009 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058011 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058014 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058016 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:31.065531 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058019 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058021 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058024 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058026 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058029 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058031 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058034 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058036 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058039 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058042 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058044 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058047 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058049 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058052 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058054 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058057 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058059 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058062 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058064 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058072 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:31.066009 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058074 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.058098 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058129 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058135 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058137 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058140 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058143 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058145 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058150 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058152 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058155 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058157 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058160 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058162 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058165 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058168 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:31.066473 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058170 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058173 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058175 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058178 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058180 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058183 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058186 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058188 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058191 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058193 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058196 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058198 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058201 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058203 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058206 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058209 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058216 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058219 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058221 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058224 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:31.066861 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058227 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058230 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058233 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058235 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058240 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058242 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058245 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058248 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058250 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058253 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058256 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058258 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058261 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058264 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058267 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058269 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058272 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058274 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058277 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058280 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:31.067390 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058282 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058285 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058288 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058290 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058293 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058295 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058298 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058301 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058303 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058311 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058314 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058317 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058320 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058322 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058325 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058327 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058331 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058333 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058336 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058338 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:31.067860 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058341 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058344 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058346 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058349 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058351 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058354 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058356 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058359 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058361 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058364 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058366 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058369 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058371 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058374 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058376 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058379 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058381 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058384 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058387 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058389 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:31.068333 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058392 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058396 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058405 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058409 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058413 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058415 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058419 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058421 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058425 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058428 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058430 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:31.058433 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.058485 2575 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.061344 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.062330 2575 server.go:1014] "Starting client certificate rotation" Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.062432 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.062470 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:51:31.068819 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.064231 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:51:31.069549 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.069535 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:51:31.080068 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.080051 2575 log.go:25] "Validated CRI v1 runtime API" Nov 02 19:51:31.083305 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.083289 2575 log.go:25] "Validated CRI v1 image API" Nov 02 19:51:31.084183 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.084158 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 19:51:31.088779 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.088754 2575 fs.go:135] Filesystem UUIDs: map[0667c6c9-0462-4dfd-afc2-998be80ed013:/dev/nvme0n1p4 6135173a-d3d7-4366-a317-e20d1c9719ea:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 02 19:51:31.088838 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.088777 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 19:51:31.094425 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.094317 2575 manager.go:217] Machine: {Timestamp:2026-11-02 19:51:31.093084047 +0000 UTC m=+0.322051991 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100594 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d99ecaa85c0424ce058bd9a9ddba6 SystemUUID:ec2d99ec-aa85-c042-4ce0-58bd9a9ddba6 BootID:bd7e8f4a-b11a-4480-b779-d12b34fa6dd1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:fa:1d:fa:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:fa:1d:fa:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:7c:6e:23:a3:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 19:51:31.094505 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.094427 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 19:51:31.094534 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.094505 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 19:51:31.095570 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.095549 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 19:51:31.095705 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.095572 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-194.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 19:51:31.095751 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.095707 2575 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 19:51:31.095751 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.095716 2575 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 19:51:31.095751 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.095750 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:51:31.095823 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.095767 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:51:31.096577 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.096567 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 19:51:31.100619 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.100607 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 19:51:31.100726 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.100717 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 19:51:31.101434 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.101425 2575 kubelet.go:493] "Attempting to sync node with API server" Nov 02 19:51:31.101471 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.101438 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 19:51:31.101471 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.101449 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 19:51:31.101471 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.101460 2575 kubelet.go:405] "Adding apiserver pod source" Nov 02 19:51:31.101471 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.101468 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 19:51:31.101853 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.101833 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:51:31.105330 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.105304 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 19:51:31.109704 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.109691 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:51:31.110401 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.110388 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 19:51:31.110440 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.110425 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 19:51:31.110615 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.110604 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 19:51:31.110646 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.110620 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 19:51:31.110646 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.110627 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 19:51:31.110646 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.110633 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 19:51:31.110646 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.110638 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 19:51:31.110646 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.110644 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 19:51:31.111226 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.111213 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 19:51:31.111255 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.111231 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 19:51:31.111255 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.111243 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 19:51:31.111255 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.111253 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 19:51:31.111332 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.111266 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 19:51:31.111332 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.111281 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 19:51:31.111768 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.111757 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 19:51:31.111800 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.111772 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 19:51:31.115406 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.115394 2575 server.go:1263] "Started kubelet" Nov 02 19:51:31.115517 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.115483 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 19:51:31.115600 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.115492 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 19:51:31.115600 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.115559 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 19:51:31.116139 ip-10-0-134-194 systemd[1]: Started Kubernetes Kubelet. Nov 02 19:51:31.116333 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.116317 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 19:51:31.119857 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.119831 2575 server.go:317] "Adding debug handlers to kubelet server" Nov 02 19:51:31.121974 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.121803 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 19:51:31.122264 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.122246 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 19:51:31.122469 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.122440 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 19:51:31.122563 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.122495 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:31.122999 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.122908 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 19:51:31.122999 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.122916 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 19:51:31.122999 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.122924 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 19:51:31.123146 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.123042 2575 reconstruct.go:97] "Volume reconstruction finished" Nov 02 19:51:31.123146 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.123050 2575 reconciler.go:29] "Reconciler: start to sync state" Nov 02 19:51:31.123245 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.123226 2575 factory.go:55] Registering systemd factory Nov 02 19:51:31.123245 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.123247 2575 factory.go:223] Registration of the systemd container factory successfully Nov 02 19:51:31.123482 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.123470 2575 factory.go:153] Registering CRI-O factory Nov 02 19:51:31.123482 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.123483 2575 factory.go:223] Registration of the crio container factory successfully Nov 02 19:51:31.123556 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.123540 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 19:51:31.123586 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.123570 2575 factory.go:103] Registering Raw factory Nov 02 19:51:31.123626 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.123589 2575 manager.go:1199] Started watching for new ooms in manager Nov 02 19:51:31.124502 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.124488 2575 manager.go:319] Starting recovery of all containers Nov 02 19:51:31.137027 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.137007 2575 manager.go:324] Recovery completed Nov 02 19:51:31.141619 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.141606 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:31.143453 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.143438 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:31.143520 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.143471 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:31.143520 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.143486 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:31.144075 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.144061 2575 cpu_manager.go:226] "Starting" policy="none" Nov 02 19:51:31.144075 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.144073 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 19:51:31.144161 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.144090 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 19:51:31.144161 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.144066 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:31.144830 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.144815 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:31.144933 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.144844 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:31.144933 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.144857 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:31.146288 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.146270 2575 policy_none.go:50] "Start" Nov 02 19:51:31.146288 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.146287 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 19:51:31.146421 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.146297 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 19:51:31.148334 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.148319 2575 policy_none.go:44] "Start" Nov 02 19:51:31.172631 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.172597 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 19:51:31.176808 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.176793 2575 manager.go:341] "Starting Device Plugin manager" Nov 02 19:51:31.176894 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.176823 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 19:51:31.176894 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.176832 2575 server.go:87] "Starting device plugin registration server" Nov 02 19:51:31.177136 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.177123 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 19:51:31.177168 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.177139 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 19:51:31.177553 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.177533 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 19:51:31.177625 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.177616 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 19:51:31.177625 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.177623 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 19:51:31.178169 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.178150 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 19:51:31.178282 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.178203 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:31.246509 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.246473 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-194.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 19:51:31.254661 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.254630 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 19:51:31.255891 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.255855 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 19:51:31.255983 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.255896 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 19:51:31.255983 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.255914 2575 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 19:51:31.255983 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.255946 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 19:51:31.278087 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.278066 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:31.278922 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.278905 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:31.279011 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.278935 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:31.279011 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.278948 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:31.279011 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.278978 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.292632 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.292582 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-194.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 19:51:31.295779 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.292456 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-194.us-west-2.compute.internal.18e452585fa23163 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-194.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-194.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:51:31.115364707 +0000 UTC m=+0.344332650,LastTimestamp:2026-11-02 19:51:31.115364707 +0000 UTC m=+0.344332650,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-194.us-west-2.compute.internal,}" Nov 02 19:51:31.307438 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.307413 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 19:51:31.307438 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.307417 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-194.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 19:51:31.356818 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.356644 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal"] Nov 02 19:51:31.356970 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.356853 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:31.357808 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.357790 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:31.357919 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.357823 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:31.357919 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.357837 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:31.359829 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.359813 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:31.359979 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.359964 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.360027 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.359996 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:31.360915 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.360897 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:31.361010 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.360937 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:31.361010 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.360948 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:31.361010 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.360899 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:31.361010 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.360988 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:31.361010 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.360998 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:31.363096 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.363079 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.363154 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.363108 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:31.363934 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.363918 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:31.363999 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.363952 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:31.363999 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.363966 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:31.373547 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.373530 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 19:51:31.373596 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.373583 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ptwd8" Nov 02 19:51:31.391090 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.391066 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 19:51:31.391187 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.391049 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-194.us-west-2.compute.internal.18e45258614ed299 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-194.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-194.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-194.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:51:31.143455385 +0000 UTC m=+0.372423329,LastTimestamp:2026-11-02 19:51:31.143455385 +0000 UTC m=+0.372423329,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-194.us-west-2.compute.internal,}" Nov 02 19:51:31.394627 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.394606 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" node="ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.397114 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.397098 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" node="ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.455973 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.455943 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:51:31.456375 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.456350 2575 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-134-194.us-west-2.compute.internal\" not found" node="ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.524750 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.524726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/770dce62893bdbeaf2564d856d57a2e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"770dce62893bdbeaf2564d856d57a2e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.524750 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.524753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/770dce62893bdbeaf2564d856d57a2e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"770dce62893bdbeaf2564d856d57a2e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.524966 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.524771 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0470b607b1bfee936d3bbd33e3118f8d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"0470b607b1bfee936d3bbd33e3118f8d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.546170 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.546117 2575 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-194.us-west-2.compute.internal\" not found" node="ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.624954 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.624921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/770dce62893bdbeaf2564d856d57a2e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"770dce62893bdbeaf2564d856d57a2e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.624954 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.624954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/770dce62893bdbeaf2564d856d57a2e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"770dce62893bdbeaf2564d856d57a2e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.625141 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.624973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0470b607b1bfee936d3bbd33e3118f8d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"0470b607b1bfee936d3bbd33e3118f8d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.625141 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.625006 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/770dce62893bdbeaf2564d856d57a2e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"770dce62893bdbeaf2564d856d57a2e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.625141 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.625016 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/770dce62893bdbeaf2564d856d57a2e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"770dce62893bdbeaf2564d856d57a2e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.625141 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.625042 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0470b607b1bfee936d3bbd33e3118f8d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"0470b607b1bfee936d3bbd33e3118f8d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.657030 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.656994 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:31.657984 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.657963 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:31.658096 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.657992 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:31.658096 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.658002 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:31.658096 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.658032 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.698070 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.698044 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.700200 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.700172 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.712099 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:31.712083 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:31.712187 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.712105 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-194.us-west-2.compute.internal\": node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:31.822537 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.822353 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:31.922591 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:31.922553 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:32.023496 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:32.023461 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:32.061987 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.061948 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 19:51:32.062412 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.062083 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 19:51:32.062412 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:32.062163 2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a83af7d525ec44176aabcd0e4302dda2-d27cf1d047af92e6.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-194.us-west-2.compute.internal.18e45258614f283e\": read tcp 10.0.134.194:56176->13.184.47.174:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-194.us-west-2.compute.internal.18e45258614f283e default 5422 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-194.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-194.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-194.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:51:31 +0000 UTC,LastTimestamp:2026-11-02 19:51:31.357829434 +0000 UTC m=+0.586797380,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-194.us-west-2.compute.internal,}" Nov 02 19:51:32.124351 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:32.124320 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:32.225538 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:32.225496 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:32.326726 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:32.326692 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:32.355240 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.355209 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:32.356276 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.356254 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:32.356391 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.356292 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:32.356391 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.356307 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:32.378635 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.378573 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:46:32 +0000 UTC" deadline="2028-05-04 10:38:59.72531216 +0000 UTC" Nov 02 19:51:32.378635 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.378602 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13166h47m27.346713356s" Nov 02 19:51:32.427682 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:32.427660 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:32.528674 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:32.528642 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:32.554404 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.554375 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:51:32.629849 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:32.629780 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:32.638235 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.638214 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:51:32.730253 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:32.730214 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:32.786218 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:32.786168 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770dce62893bdbeaf2564d856d57a2e6.slice/crio-73f662b539aae6ba3dc20130eaddfea9caf3ca04e7fd2c4d52b6ad2477be4367 WatchSource:0}: Error finding container 73f662b539aae6ba3dc20130eaddfea9caf3ca04e7fd2c4d52b6ad2477be4367: Status 404 returned error can't find the container with id 73f662b539aae6ba3dc20130eaddfea9caf3ca04e7fd2c4d52b6ad2477be4367 Nov 02 19:51:32.793738 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.793711 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:51:32.799760 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:32.799734 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0470b607b1bfee936d3bbd33e3118f8d.slice/crio-02fc097a077a44cf61b61260e0a9c49039a5fb31363458ad74e8dd86c890b543 WatchSource:0}: Error finding container 02fc097a077a44cf61b61260e0a9c49039a5fb31363458ad74e8dd86c890b543: Status 404 returned error can't find the container with id 02fc097a077a44cf61b61260e0a9c49039a5fb31363458ad74e8dd86c890b543 Nov 02 19:51:32.830473 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:32.830440 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:51:32.839922 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.839885 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:51:32.923008 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.922913 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:32.984351 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.984315 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:51:32.985514 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:32.985498 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:51:33.043826 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.043797 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:51:33.102940 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.102896 2575 apiserver.go:52] "Watching apiserver" Nov 02 19:51:33.122205 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.122177 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 19:51:33.206684 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.206599 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 19:51:33.206988 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.206967 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-m2tvk","kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-nqhf8","openshift-ovn-kubernetes/ovnkube-node-r6lg7","kube-system/global-pull-secret-syncer-4bpmh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal","kube-system/konnectivity-agent-r9jxd","openshift-dns/node-resolver-9rglx","openshift-multus/multus-z2vn7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk","openshift-cluster-node-tuning-operator/tuned-7m9qt","openshift-image-registry/node-ca-dsct6","openshift-multus/multus-additional-cni-plugins-7svjs","openshift-multus/network-metrics-daemon-r67nn"] Nov 02 19:51:33.210791 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.210774 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:51:33.213163 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.213140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.215540 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.215524 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:51:33.218088 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.217932 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.220159 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.220138 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:33.220263 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.220238 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:33.223473 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.223452 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:33.223666 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.223525 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:33.225965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.225934 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.228223 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.228205 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:33.228305 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.228255 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:33.230462 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.230443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9rglx" Nov 02 19:51:33.232627 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.232607 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.232886 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.232865 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-system-cni-dir\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.232979 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.232902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-run-netns\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233045 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-cnibin\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.233045 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.233165 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c10c2efe-3e34-4ca7-b972-680cde439609-iptables-alerter-script\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:51:33.233165 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-node-log\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233165 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233094 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-env-overrides\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233165 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233122 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6dde0e70-73a7-4198-831e-875401c4c3cd-dbus\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:33.233165 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233422 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-cni-bin\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233422 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-cni-netd\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233422 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233422 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:33.233422 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.233422 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-slash\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233422 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-ovn\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233422 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.233422 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-kubelet\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233422 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233417 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-log-socket\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233457 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98798482-f8b5-401e-9aa2-363cda917fba-ovn-node-metrics-cert\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-ovnkube-script-lib\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233520 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-systemd-units\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-systemd\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4j9w\" (UniqueName: \"kubernetes.io/projected/0d60e019-846f-497e-a1fb-1c972295523b-kube-api-access-s4j9w\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233692 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54ff0392-2388-465d-94b5-c2839f6eac84-agent-certs\") pod \"konnectivity-agent-r9jxd\" (UID: \"54ff0392-2388-465d-94b5-c2839f6eac84\") " pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54ff0392-2388-465d-94b5-c2839f6eac84-konnectivity-ca\") pod \"konnectivity-agent-r9jxd\" (UID: \"54ff0392-2388-465d-94b5-c2839f6eac84\") " pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233773 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgkrt\" (UniqueName: \"kubernetes.io/projected/a3823ef4-9aae-417c-bc9a-a04253589bf5-kube-api-access-fgkrt\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233818 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v7m7\" (UniqueName: \"kubernetes.io/projected/c10c2efe-3e34-4ca7-b972-680cde439609-kube-api-access-4v7m7\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-etc-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233898 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-ovnkube-config\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.233965 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233967 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6dde0e70-73a7-4198-831e-875401c4c3cd-kubelet-config\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:33.234549 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.233993 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c10c2efe-3e34-4ca7-b972-680cde439609-host-slash\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:51:33.234549 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.234031 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-var-lib-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.234549 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.234050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb5kf\" (UniqueName: \"kubernetes.io/projected/98798482-f8b5-401e-9aa2-363cda917fba-kube-api-access-bb5kf\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.235054 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.235039 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.238058 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.237643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:51:33.240364 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.238303 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:51:33.260678 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.260639 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" event={"ID":"0470b607b1bfee936d3bbd33e3118f8d","Type":"ContainerStarted","Data":"02fc097a077a44cf61b61260e0a9c49039a5fb31363458ad74e8dd86c890b543"} Nov 02 19:51:33.261874 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.261853 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" event={"ID":"770dce62893bdbeaf2564d856d57a2e6","Type":"ContainerStarted","Data":"73f662b539aae6ba3dc20130eaddfea9caf3ca04e7fd2c4d52b6ad2477be4367"} Nov 02 19:51:33.263282 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.263257 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 19:51:33.263455 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.263439 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 19:51:33.263506 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.263446 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f8f9p\"" Nov 02 19:51:33.265248 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.265229 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 19:51:33.265336 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.265287 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 19:51:33.265336 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.265315 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 19:51:33.265336 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.265319 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 19:51:33.265452 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.265333 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 19:51:33.265452 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.265366 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8m7zt\"" Nov 02 19:51:33.267434 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.267413 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:51:33.267568 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.267543 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 19:51:33.267622 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.267582 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 19:51:33.267622 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.267599 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4w4l2\"" Nov 02 19:51:33.269894 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.269877 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlz48\"" Nov 02 19:51:33.270032 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.270012 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 19:51:33.270131 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.270089 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 19:51:33.270182 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.270167 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 19:51:33.270260 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.270249 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 19:51:33.270408 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.270395 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 19:51:33.270497 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.270486 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 19:51:33.277823 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.277809 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 19:51:33.324471 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.324444 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 19:51:33.335226 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-node-log\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335320 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-env-overrides\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335320 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-cni-bin\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.335320 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335270 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-kubelet\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.335320 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335311 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-node-log\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335538 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87tb\" (UniqueName: \"kubernetes.io/projected/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-kube-api-access-c87tb\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.335538 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-device-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.335538 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335538 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-cni-bin\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335538 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335476 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335538 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-sys-fs\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.335538 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-modprobe-d\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.335538 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335529 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysconfig\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335554 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-cni-bin\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-systemd\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335579 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335591 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-tuned\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-slash\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-ovn\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-cni-multus\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335707 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-slash\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335714 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-ovn\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysctl-d\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-var-lib-kubelet\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-kubelet\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-log-socket\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335807 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-kubelet\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98798482-f8b5-401e-9aa2-363cda917fba-ovn-node-metrics-cert\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.335961 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-cni-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335852 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-daemon-config\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335851 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-log-socket\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335868 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-env-overrides\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335908 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-systemd-units\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335951 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnwzr\" (UniqueName: \"kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr\") pod \"network-check-target-m2tvk\" (UID: \"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9\") " pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4j9w\" (UniqueName: \"kubernetes.io/projected/0d60e019-846f-497e-a1fb-1c972295523b-kube-api-access-s4j9w\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335973 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-systemd-units\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.335984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54ff0392-2388-465d-94b5-c2839f6eac84-agent-certs\") pod \"konnectivity-agent-r9jxd\" (UID: \"54ff0392-2388-465d-94b5-c2839f6eac84\") " pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.335986 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54ff0392-2388-465d-94b5-c2839f6eac84-konnectivity-ca\") pod \"konnectivity-agent-r9jxd\" (UID: \"54ff0392-2388-465d-94b5-c2839f6eac84\") " pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.336050 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret podName:6dde0e70-73a7-4198-831e-875401c4c3cd nodeName:}" failed. No retries permitted until 2026-11-02 19:51:33.836029503 +0000 UTC m=+3.064997453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret") pod "global-pull-secret-syncer-4bpmh" (UID: "6dde0e70-73a7-4198-831e-875401c4c3cd") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7m7\" (UniqueName: \"kubernetes.io/projected/c10c2efe-3e34-4ca7-b972-680cde439609-kube-api-access-4v7m7\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-etc-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-ovnkube-config\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-registration-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.336789 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336199 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-etc-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336215 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-sys\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-etc-kubernetes\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-var-lib-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336359 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb5kf\" (UniqueName: \"kubernetes.io/projected/98798482-f8b5-401e-9aa2-363cda917fba-kube-api-access-bb5kf\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336389 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-system-cni-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336391 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-var-lib-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336417 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-lib-modules\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-run-netns\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4j5s\" (UniqueName: \"kubernetes.io/projected/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-kube-api-access-s4j5s\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-cnibin\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336562 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-hostroot\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336610 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e588ae83-565a-4711-a7b7-d77b8ae24bc8-tmp-dir\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336626 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54ff0392-2388-465d-94b5-c2839f6eac84-konnectivity-ca\") pod \"konnectivity-agent-r9jxd\" (UID: \"54ff0392-2388-465d-94b5-c2839f6eac84\") " pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-run\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336627 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-run-netns\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.337525 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336667 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-ovnkube-config\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336675 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0ed3429-8e54-4f88-b820-751eacadd01a-tmp\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336669 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-cnibin\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6dde0e70-73a7-4198-831e-875401c4c3cd-dbus\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336743 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-cni-netd\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-cni-binary-copy\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336791 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-cni-netd\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336811 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-k8s-cni-cncf-io\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336859 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6dde0e70-73a7-4198-831e-875401c4c3cd-dbus\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxbrc\" (UniqueName: \"kubernetes.io/projected/f0ed3429-8e54-4f88-b820-751eacadd01a-kube-api-access-cxbrc\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.336982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.336995 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.338371 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.337045 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs podName:0d60e019-846f-497e-a1fb-1c972295523b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:33.837027581 +0000 UTC m=+3.065995537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs") pod "network-metrics-daemon-r67nn" (UID: "0d60e019-846f-497e-a1fb-1c972295523b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-ovnkube-script-lib\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-multus-certs\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46jj\" (UniqueName: \"kubernetes.io/projected/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-kube-api-access-q46jj\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337182 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337347 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-systemd\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337380 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-kubernetes\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-host\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-serviceca\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337505 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgkrt\" (UniqueName: \"kubernetes.io/projected/a3823ef4-9aae-417c-bc9a-a04253589bf5-kube-api-access-fgkrt\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337532 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337571 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-cnibin\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-netns\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337615 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6dde0e70-73a7-4198-831e-875401c4c3cd-kubelet-config\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337673 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c10c2efe-3e34-4ca7-b972-680cde439609-host-slash\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:51:33.339107 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337703 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-systemd\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337720 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-ovnkube-script-lib\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337779 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337790 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6dde0e70-73a7-4198-831e-875401c4c3cd-kubelet-config\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337827 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c10c2efe-3e34-4ca7-b972-680cde439609-host-slash\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337824 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-socket-dir-parent\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e588ae83-565a-4711-a7b7-d77b8ae24bc8-hosts-file\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysctl-conf\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.337997 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-host\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.338026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-system-cni-dir\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.338068 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-conf-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.338098 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-system-cni-dir\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.338109 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppgng\" (UniqueName: \"kubernetes.io/projected/e588ae83-565a-4711-a7b7-d77b8ae24bc8-kube-api-access-ppgng\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.338160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-socket-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.338196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.338228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c10c2efe-3e34-4ca7-b972-680cde439609-iptables-alerter-script\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:51:33.339612 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.338383 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.340365 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.338796 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c10c2efe-3e34-4ca7-b972-680cde439609-iptables-alerter-script\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:51:33.340365 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.339937 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98798482-f8b5-401e-9aa2-363cda917fba-ovn-node-metrics-cert\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.340365 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.340167 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54ff0392-2388-465d-94b5-c2839f6eac84-agent-certs\") pod \"konnectivity-agent-r9jxd\" (UID: \"54ff0392-2388-465d-94b5-c2839f6eac84\") " pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:51:33.367562 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.367537 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 19:51:33.367668 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.367630 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t5l8f" Nov 02 19:51:33.373339 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373175 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 19:51:33.373339 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373207 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 19:51:33.373339 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373222 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 19:51:33.373339 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373212 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 19:51:33.373339 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373209 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vl78h\"" Nov 02 19:51:33.373339 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373275 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 19:51:33.373339 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373279 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 19:51:33.373339 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373303 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 19:51:33.373719 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373375 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dbl8w\"" Nov 02 19:51:33.373719 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373393 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:51:33.373719 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373436 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 19:51:33.373719 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373457 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4hfb\"" Nov 02 19:51:33.373719 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373496 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-shcvr\"" Nov 02 19:51:33.373719 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373562 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qqgmv\"" Nov 02 19:51:33.373719 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.373612 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 19:51:33.393510 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.393482 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4j9w\" (UniqueName: \"kubernetes.io/projected/0d60e019-846f-497e-a1fb-1c972295523b-kube-api-access-s4j9w\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:33.393642 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.393625 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb5kf\" (UniqueName: \"kubernetes.io/projected/98798482-f8b5-401e-9aa2-363cda917fba-kube-api-access-bb5kf\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.393778 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.393761 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7m7\" (UniqueName: \"kubernetes.io/projected/c10c2efe-3e34-4ca7-b972-680cde439609-kube-api-access-4v7m7\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:51:33.393843 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.393828 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgkrt\" (UniqueName: \"kubernetes.io/projected/a3823ef4-9aae-417c-bc9a-a04253589bf5-kube-api-access-fgkrt\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.438692 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.438644 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-hostroot\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.438809 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.438704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e588ae83-565a-4711-a7b7-d77b8ae24bc8-tmp-dir\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:51:33.438809 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.438744 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-run\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.438809 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.438751 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-hostroot\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.438809 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.438776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0ed3429-8e54-4f88-b820-751eacadd01a-tmp\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.439378 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-cni-binary-copy\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.439533 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439381 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-run\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.439533 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-k8s-cni-cncf-io\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.439533 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439458 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e588ae83-565a-4711-a7b7-d77b8ae24bc8-tmp-dir\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:51:33.439533 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.439832 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbrc\" (UniqueName: \"kubernetes.io/projected/f0ed3429-8e54-4f88-b820-751eacadd01a-kube-api-access-cxbrc\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.439832 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439632 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.439832 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.439832 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439680 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-k8s-cni-cncf-io\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.439832 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439697 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-multus-certs\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.439832 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46jj\" (UniqueName: \"kubernetes.io/projected/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-kube-api-access-q46jj\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:51:33.439832 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-kubernetes\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.440347 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-host\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:51:33.440347 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439875 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-multus-certs\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.440347 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.439892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-serviceca\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:51:33.440347 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.440041 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.440347 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.440103 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-kubernetes\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.440347 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.440156 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-cni-binary-copy\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.440347 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.440259 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-host\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:51:33.440808 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.440409 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-serviceca\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:51:33.440808 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.440473 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-cnibin\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.440808 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.440522 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-cnibin\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.441658 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-netns\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.441791 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-socket-dir-parent\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.441791 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e588ae83-565a-4711-a7b7-d77b8ae24bc8-hosts-file\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:51:33.441791 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysctl-conf\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.441791 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-host\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.442101 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441821 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0ed3429-8e54-4f88-b820-751eacadd01a-tmp\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.442101 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441828 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-socket-dir-parent\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.442101 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441861 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-netns\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.442101 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-conf-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.442101 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441892 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-conf-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.442101 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppgng\" (UniqueName: \"kubernetes.io/projected/e588ae83-565a-4711-a7b7-d77b8ae24bc8-kube-api-access-ppgng\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:51:33.442101 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441935 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-host\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.442101 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-socket-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.442101 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.441935 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e588ae83-565a-4711-a7b7-d77b8ae24bc8-hosts-file\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:51:33.442101 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442096 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysctl-conf\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.442101 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442095 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-socket-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-cni-bin\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-kubelet\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87tb\" (UniqueName: \"kubernetes.io/projected/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-kube-api-access-c87tb\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442301 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-kubelet\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442338 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-cni-bin\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-device-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-sys-fs\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-modprobe-d\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442478 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysconfig\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442507 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-systemd\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-tuned\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-cni-multus\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442599 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysctl-d\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-var-lib-kubelet\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442636 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-modprobe-d\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442657 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-cni-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442688 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-device-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.442852 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-daemon-config\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442814 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwzr\" (UniqueName: \"kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr\") pod \"network-check-target-m2tvk\" (UID: \"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9\") " pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-registration-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-sys\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-etc-kubernetes\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-system-cni-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.442996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-lib-modules\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4j5s\" (UniqueName: \"kubernetes.io/projected/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-kube-api-access-s4j5s\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443242 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-cni-multus\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443259 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysconfig\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443300 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-sys\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443382 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-cni-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443384 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-system-cni-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443483 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-etc-kubernetes\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443508 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-registration-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443550 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-systemd\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443561 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-sys-fs\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443624 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysctl-d\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.443941 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443637 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-var-lib-kubelet\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.444858 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.443899 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-lib-modules\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.444858 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.444432 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-daemon-config\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.445036 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.445019 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-tuned\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.496469 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.496412 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbrc\" (UniqueName: \"kubernetes.io/projected/f0ed3429-8e54-4f88-b820-751eacadd01a-kube-api-access-cxbrc\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.497038 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.497005 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46jj\" (UniqueName: \"kubernetes.io/projected/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-kube-api-access-q46jj\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:51:33.497927 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.497904 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:33.498011 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.497932 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:33.498011 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.497950 2575 projected.go:196] Error preparing data for projected volume kube-api-access-gnwzr for pod openshift-network-diagnostics/network-check-target-m2tvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:33.498085 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.498038 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr podName:b2ff0bc2-a4cf-45f3-b870-2526d024b6c9 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:33.99801773 +0000 UTC m=+3.226985687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gnwzr" (UniqueName: "kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr") pod "network-check-target-m2tvk" (UID: "b2ff0bc2-a4cf-45f3-b870-2526d024b6c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:33.498666 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.498643 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppgng\" (UniqueName: \"kubernetes.io/projected/e588ae83-565a-4711-a7b7-d77b8ae24bc8-kube-api-access-ppgng\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:51:33.498838 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.498822 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87tb\" (UniqueName: \"kubernetes.io/projected/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-kube-api-access-c87tb\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.499460 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.499435 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4j5s\" (UniqueName: \"kubernetes.io/projected/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-kube-api-access-s4j5s\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.523533 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.523502 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:51:33.534233 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:33.534211 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ff0392_2388_465d_94b5_c2839f6eac84.slice/crio-3d39bbb7bf19d99d8016b542db8d07a6d84fb326f5c9f5610a1768ccca21a035 WatchSource:0}: Error finding container 3d39bbb7bf19d99d8016b542db8d07a6d84fb326f5c9f5610a1768ccca21a035: Status 404 returned error can't find the container with id 3d39bbb7bf19d99d8016b542db8d07a6d84fb326f5c9f5610a1768ccca21a035 Nov 02 19:51:33.543137 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.543115 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:51:33.547011 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.546998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:51:33.552039 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:33.552015 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10c2efe_3e34_4ca7_b972_680cde439609.slice/crio-b8cd86680e0044abba406a9e8aaa8f180955fdbcb150e85487e7246c730f853f WatchSource:0}: Error finding container b8cd86680e0044abba406a9e8aaa8f180955fdbcb150e85487e7246c730f853f: Status 404 returned error can't find the container with id b8cd86680e0044abba406a9e8aaa8f180955fdbcb150e85487e7246c730f853f Nov 02 19:51:33.553880 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.553867 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z2vn7" Nov 02 19:51:33.556368 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.556344 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9rglx" Nov 02 19:51:33.556685 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:33.556660 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3823ef4_9aae_417c_bc9a_a04253589bf5.slice/crio-31198e0b2a84a17c7d0d20a0ff5f15365a71d4ef25070fea99b96b564632a206 WatchSource:0}: Error finding container 31198e0b2a84a17c7d0d20a0ff5f15365a71d4ef25070fea99b96b564632a206: Status 404 returned error can't find the container with id 31198e0b2a84a17c7d0d20a0ff5f15365a71d4ef25070fea99b96b564632a206 Nov 02 19:51:33.564571 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:33.564550 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5d5ddb_65f0_4e9d_b1f3_7115a1087e57.slice/crio-ab5d393663b3d2aefb1aeaa801d18001114cadd1d6df0260dd98f18de4fdc46c WatchSource:0}: Error finding container ab5d393663b3d2aefb1aeaa801d18001114cadd1d6df0260dd98f18de4fdc46c: Status 404 returned error can't find the container with id ab5d393663b3d2aefb1aeaa801d18001114cadd1d6df0260dd98f18de4fdc46c Nov 02 19:51:33.565196 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:33.565172 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode588ae83_565a_4711_a7b7_d77b8ae24bc8.slice/crio-0a2a3a7bb67cda384d06d04ecde62c7e5e7c574c2712ebc9f2798a5fae4aac8e WatchSource:0}: Error finding container 0a2a3a7bb67cda384d06d04ecde62c7e5e7c574c2712ebc9f2798a5fae4aac8e: Status 404 returned error can't find the container with id 0a2a3a7bb67cda384d06d04ecde62c7e5e7c574c2712ebc9f2798a5fae4aac8e Nov 02 19:51:33.572249 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.572232 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:33.574514 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.574499 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:51:33.576182 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.576166 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:51:33.578264 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.578248 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:51:33.582204 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:33.582183 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98798482_f8b5_401e_9aa2_363cda917fba.slice/crio-5f7105b2ce2bca00f516d0fcc551248c1ffada61b655810271ee923b37ea20ad WatchSource:0}: Error finding container 5f7105b2ce2bca00f516d0fcc551248c1ffada61b655810271ee923b37ea20ad: Status 404 returned error can't find the container with id 5f7105b2ce2bca00f516d0fcc551248c1ffada61b655810271ee923b37ea20ad Nov 02 19:51:33.585840 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:33.585819 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ed3429_8e54_4f88_b820_751eacadd01a.slice/crio-022c57e776f8c45bb9d751ea621eca09cdac440cc4d94892f08bb5b1e7cbd8b3 WatchSource:0}: Error finding container 022c57e776f8c45bb9d751ea621eca09cdac440cc4d94892f08bb5b1e7cbd8b3: Status 404 returned error can't find the container with id 022c57e776f8c45bb9d751ea621eca09cdac440cc4d94892f08bb5b1e7cbd8b3 Nov 02 19:51:33.589331 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:33.589307 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10bb7fa_2eb9_4215_b467_c2165c4c76f5.slice/crio-ca8189f849c255c459632645fbb1e2159d535964b6fc61524994834c4b1f35df WatchSource:0}: Error finding container ca8189f849c255c459632645fbb1e2159d535964b6fc61524994834c4b1f35df: Status 404 returned error can't find the container with id ca8189f849c255c459632645fbb1e2159d535964b6fc61524994834c4b1f35df Nov 02 19:51:33.592247 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:51:33.592226 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6f9fda_5459_4d5a_8405_7eab0de2cb1e.slice/crio-9972847e89925e9621b618b0a2bfd6449cd0e69e9cbc206944b2bcbc0cec80c3 WatchSource:0}: Error finding container 9972847e89925e9621b618b0a2bfd6449cd0e69e9cbc206944b2bcbc0cec80c3: Status 404 returned error can't find the container with id 9972847e89925e9621b618b0a2bfd6449cd0e69e9cbc206944b2bcbc0cec80c3 Nov 02 19:51:33.846338 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.846251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:33.846338 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:33.846316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:33.846552 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.846440 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:33.846552 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.846510 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret podName:6dde0e70-73a7-4198-831e-875401c4c3cd nodeName:}" failed. No retries permitted until 2026-11-02 19:51:34.846492266 +0000 UTC m=+4.075460215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret") pod "global-pull-secret-syncer-4bpmh" (UID: "6dde0e70-73a7-4198-831e-875401c4c3cd") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:33.846552 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.846440 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:33.846823 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:33.846567 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs podName:0d60e019-846f-497e-a1fb-1c972295523b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:34.846556 +0000 UTC m=+4.075523940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs") pod "network-metrics-daemon-r67nn" (UID: "0d60e019-846f-497e-a1fb-1c972295523b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:34.047558 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.047510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwzr\" (UniqueName: \"kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr\") pod \"network-check-target-m2tvk\" (UID: \"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9\") " pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:34.047819 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:34.047666 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:34.047819 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:34.047693 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:34.047819 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:34.047707 2575 projected.go:196] Error preparing data for projected volume kube-api-access-gnwzr for pod openshift-network-diagnostics/network-check-target-m2tvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:34.047819 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:34.047775 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr podName:b2ff0bc2-a4cf-45f3-b870-2526d024b6c9 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:35.0477528 +0000 UTC m=+4.276720751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnwzr" (UniqueName: "kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr") pod "network-check-target-m2tvk" (UID: "b2ff0bc2-a4cf-45f3-b870-2526d024b6c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:34.265072 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.264983 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nqhf8" event={"ID":"c10c2efe-3e34-4ca7-b972-680cde439609","Type":"ContainerStarted","Data":"b8cd86680e0044abba406a9e8aaa8f180955fdbcb150e85487e7246c730f853f"} Nov 02 19:51:34.266352 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.266327 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" event={"ID":"d10bb7fa-2eb9-4215-b467-c2165c4c76f5","Type":"ContainerStarted","Data":"ca8189f849c255c459632645fbb1e2159d535964b6fc61524994834c4b1f35df"} Nov 02 19:51:34.267796 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.267770 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" event={"ID":"f0ed3429-8e54-4f88-b820-751eacadd01a","Type":"ContainerStarted","Data":"022c57e776f8c45bb9d751ea621eca09cdac440cc4d94892f08bb5b1e7cbd8b3"} Nov 02 19:51:34.269320 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.269292 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" event={"ID":"98798482-f8b5-401e-9aa2-363cda917fba","Type":"ContainerStarted","Data":"5f7105b2ce2bca00f516d0fcc551248c1ffada61b655810271ee923b37ea20ad"} Nov 02 19:51:34.270634 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.270588 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z2vn7" event={"ID":"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57","Type":"ContainerStarted","Data":"ab5d393663b3d2aefb1aeaa801d18001114cadd1d6df0260dd98f18de4fdc46c"} Nov 02 19:51:34.271962 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.271938 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r9jxd" event={"ID":"54ff0392-2388-465d-94b5-c2839f6eac84","Type":"ContainerStarted","Data":"3d39bbb7bf19d99d8016b542db8d07a6d84fb326f5c9f5610a1768ccca21a035"} Nov 02 19:51:34.273168 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.273150 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dsct6" event={"ID":"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e","Type":"ContainerStarted","Data":"9972847e89925e9621b618b0a2bfd6449cd0e69e9cbc206944b2bcbc0cec80c3"} Nov 02 19:51:34.274470 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.274447 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9rglx" event={"ID":"e588ae83-565a-4711-a7b7-d77b8ae24bc8","Type":"ContainerStarted","Data":"0a2a3a7bb67cda384d06d04ecde62c7e5e7c574c2712ebc9f2798a5fae4aac8e"} Nov 02 19:51:34.275732 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.275710 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7svjs" event={"ID":"a3823ef4-9aae-417c-bc9a-a04253589bf5","Type":"ContainerStarted","Data":"31198e0b2a84a17c7d0d20a0ff5f15365a71d4ef25070fea99b96b564632a206"} Nov 02 19:51:34.372242 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.372143 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:46:34 +0000 UTC" deadline="2028-06-11 14:15:41.793418216 +0000 UTC" Nov 02 19:51:34.372242 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.372184 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14082h24m7.421239241s" Nov 02 19:51:34.856425 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.856378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:34.856627 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:34.856464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:34.856725 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:34.856705 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:34.856794 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:34.856780 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs podName:0d60e019-846f-497e-a1fb-1c972295523b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:36.856759395 +0000 UTC m=+6.085727333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs") pod "network-metrics-daemon-r67nn" (UID: "0d60e019-846f-497e-a1fb-1c972295523b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:34.856863 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:34.856849 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:34.856949 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:34.856885 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret podName:6dde0e70-73a7-4198-831e-875401c4c3cd nodeName:}" failed. No retries permitted until 2026-11-02 19:51:36.856873855 +0000 UTC m=+6.085841789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret") pod "global-pull-secret-syncer-4bpmh" (UID: "6dde0e70-73a7-4198-831e-875401c4c3cd") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:35.058161 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:35.058121 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwzr\" (UniqueName: \"kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr\") pod \"network-check-target-m2tvk\" (UID: \"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9\") " pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:35.058333 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:35.058310 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:35.058333 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:35.058329 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:35.058456 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:35.058343 2575 projected.go:196] Error preparing data for projected volume kube-api-access-gnwzr for pod openshift-network-diagnostics/network-check-target-m2tvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:35.058456 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:35.058417 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr podName:b2ff0bc2-a4cf-45f3-b870-2526d024b6c9 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:37.058397955 +0000 UTC m=+6.287365904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnwzr" (UniqueName: "kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr") pod "network-check-target-m2tvk" (UID: "b2ff0bc2-a4cf-45f3-b870-2526d024b6c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:35.259041 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:35.258964 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:35.259200 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:35.259126 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:35.259595 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:35.259575 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:35.259721 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:35.259699 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:35.259931 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:35.259912 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:35.260092 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:35.260008 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:35.376141 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:35.376069 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:46:34 +0000 UTC" deadline="2028-07-08 04:31:12.176370736 +0000 UTC" Nov 02 19:51:35.376141 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:35.376136 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14720h39m36.800238927s" Nov 02 19:51:36.881703 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:36.881590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:36.881703 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:36.881679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:36.882338 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:36.881737 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:36.882338 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:36.881811 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs podName:0d60e019-846f-497e-a1fb-1c972295523b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:40.881793064 +0000 UTC m=+10.110761000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs") pod "network-metrics-daemon-r67nn" (UID: "0d60e019-846f-497e-a1fb-1c972295523b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:36.882338 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:36.881842 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:36.882338 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:36.881909 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret podName:6dde0e70-73a7-4198-831e-875401c4c3cd nodeName:}" failed. No retries permitted until 2026-11-02 19:51:40.881892966 +0000 UTC m=+10.110860903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret") pod "global-pull-secret-syncer-4bpmh" (UID: "6dde0e70-73a7-4198-831e-875401c4c3cd") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:37.085308 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:37.085275 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwzr\" (UniqueName: \"kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr\") pod \"network-check-target-m2tvk\" (UID: \"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9\") " pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:37.085497 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:37.085452 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:37.085497 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:37.085476 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:37.085497 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:37.085486 2575 projected.go:196] Error preparing data for projected volume kube-api-access-gnwzr for pod openshift-network-diagnostics/network-check-target-m2tvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:37.085673 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:37.085534 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr podName:b2ff0bc2-a4cf-45f3-b870-2526d024b6c9 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:41.085518115 +0000 UTC m=+10.314486062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnwzr" (UniqueName: "kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr") pod "network-check-target-m2tvk" (UID: "b2ff0bc2-a4cf-45f3-b870-2526d024b6c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:37.257511 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:37.257057 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:37.257511 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:37.257202 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:37.257511 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:37.257265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:37.257511 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:37.257322 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:37.258106 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:37.257909 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:37.258106 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:37.257981 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:37.293326 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:37.292558 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" event={"ID":"0470b607b1bfee936d3bbd33e3118f8d","Type":"ContainerStarted","Data":"486428f7f3853e4f4f0d09ad2b3de9388e4dbe5cf19de02dd95b960aadf6f6a3"} Nov 02 19:51:37.296786 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:37.296752 2575 generic.go:356] "Generic (PLEG): container finished" podID="770dce62893bdbeaf2564d856d57a2e6" containerID="1564ef31fa65a6b0d086e94f8f790e3068003943ea75f3b122948ab5cee90ca1" exitCode=0 Nov 02 19:51:37.296938 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:37.296798 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" event={"ID":"770dce62893bdbeaf2564d856d57a2e6","Type":"ContainerDied","Data":"1564ef31fa65a6b0d086e94f8f790e3068003943ea75f3b122948ab5cee90ca1"} Nov 02 19:51:37.527650 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:37.522564 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" podStartSLOduration=4.522542026 podStartE2EDuration="4.522542026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:37.405117105 +0000 UTC m=+6.634085060" watchObservedRunningTime="2026-11-02 19:51:37.522542026 +0000 UTC m=+6.751509987" Nov 02 19:51:38.304414 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:38.303421 2575 generic.go:356] "Generic (PLEG): container finished" podID="a3823ef4-9aae-417c-bc9a-a04253589bf5" containerID="54bac71d92eb7cfc5e1ef6a38b686dc913dfffa8cd3623a45e46a001564cbd3f" exitCode=0 Nov 02 19:51:38.304414 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:38.303519 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7svjs" event={"ID":"a3823ef4-9aae-417c-bc9a-a04253589bf5","Type":"ContainerDied","Data":"54bac71d92eb7cfc5e1ef6a38b686dc913dfffa8cd3623a45e46a001564cbd3f"} Nov 02 19:51:38.318200 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:38.318167 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" event={"ID":"770dce62893bdbeaf2564d856d57a2e6","Type":"ContainerStarted","Data":"47811b61c36d347c19afca864ac647e6932e3b7032cff1995683c472381eed2f"} Nov 02 19:51:39.258565 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:39.258535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:39.258758 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:39.258667 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:39.259112 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:39.259092 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:39.259224 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:39.259199 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:39.259293 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:39.259268 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:39.259358 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:39.259346 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:39.326285 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:39.326221 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r9jxd" event={"ID":"54ff0392-2388-465d-94b5-c2839f6eac84","Type":"ContainerStarted","Data":"10855e7609a6e4008a58ebf29292ead33a7822732df1c750b9f18d225f9f07b4"} Nov 02 19:51:39.439786 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:39.438693 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" podStartSLOduration=6.43867073 podStartE2EDuration="6.43867073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:38.538520687 +0000 UTC m=+7.767488647" watchObservedRunningTime="2026-11-02 19:51:39.43867073 +0000 UTC m=+8.667638689" Nov 02 19:51:40.332417 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:40.332374 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" event={"ID":"d10bb7fa-2eb9-4215-b467-c2165c4c76f5","Type":"ContainerStarted","Data":"ec2f221a3a98d5a7008dc99ac8f9d3a3791a0dd77e8cde5bc260112740592138"} Nov 02 19:51:40.337475 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:40.337444 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dsct6" event={"ID":"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e","Type":"ContainerStarted","Data":"a46cd2f7e4c779733b09132f26fd41135a1fe8e726c6541e61a330d26158dfe9"} Nov 02 19:51:40.466347 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:40.464236 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dsct6" podStartSLOduration=2.564709397 podStartE2EDuration="8.46421579s" totalImagesPullingTime="5.899506393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:40.463608017 +0000 UTC m=+9.692575976" watchObservedRunningTime="2026-11-02 19:51:40.46421579 +0000 UTC m=+9.693183763" Nov 02 19:51:40.466347 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:40.465241 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r9jxd" podStartSLOduration=3.8631290529999998 podStartE2EDuration="8.465224347s" totalImagesPullingTime="4.602095294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:39.439752206 +0000 UTC m=+8.668720165" watchObservedRunningTime="2026-11-02 19:51:40.465224347 +0000 UTC m=+9.694192305" Nov 02 19:51:40.917784 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:40.917695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:40.917784 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:40.917778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:40.918094 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:40.917929 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:40.918094 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:40.917996 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs podName:0d60e019-846f-497e-a1fb-1c972295523b nodeName:}" failed. No retries permitted until 2026-11-02 19:51:48.917976733 +0000 UTC m=+18.146944684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs") pod "network-metrics-daemon-r67nn" (UID: "0d60e019-846f-497e-a1fb-1c972295523b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:40.918446 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:40.918430 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:40.918512 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:40.918485 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret podName:6dde0e70-73a7-4198-831e-875401c4c3cd nodeName:}" failed. No retries permitted until 2026-11-02 19:51:48.918470048 +0000 UTC m=+18.147437999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret") pod "global-pull-secret-syncer-4bpmh" (UID: "6dde0e70-73a7-4198-831e-875401c4c3cd") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:41.121629 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:41.121586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwzr\" (UniqueName: \"kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr\") pod \"network-check-target-m2tvk\" (UID: \"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9\") " pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:41.121824 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:41.121764 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:41.121824 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:41.121787 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:41.121824 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:41.121801 2575 projected.go:196] Error preparing data for projected volume kube-api-access-gnwzr for pod openshift-network-diagnostics/network-check-target-m2tvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:41.122005 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:41.121866 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr podName:b2ff0bc2-a4cf-45f3-b870-2526d024b6c9 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:49.121847571 +0000 UTC m=+18.350815510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnwzr" (UniqueName: "kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr") pod "network-check-target-m2tvk" (UID: "b2ff0bc2-a4cf-45f3-b870-2526d024b6c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:41.258226 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:41.258155 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:41.258396 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:41.258261 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:41.258595 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:41.258577 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:41.258679 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:41.258660 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:41.258746 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:41.258705 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:41.258802 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:41.258747 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:42.098641 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:42.098005 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:51:42.098641 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:42.098567 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:51:42.439051 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:42.438986 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 19:51:43.187851 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:43.187537 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:51:42.439009891Z","UUID":"77447f5a-0de6-41fa-ac29-cddda3b7d417","Handler":null,"Name":"","Endpoint":""} Nov 02 19:51:43.189149 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:43.189087 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 19:51:43.189235 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:43.189162 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 19:51:43.258393 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:43.258363 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:43.258569 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:43.258487 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:43.258628 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:43.258574 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:43.258679 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:43.258649 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:43.258729 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:43.258715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:43.258791 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:43.258773 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:43.349578 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:43.349541 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9rglx" event={"ID":"e588ae83-565a-4711-a7b7-d77b8ae24bc8","Type":"ContainerStarted","Data":"076d2c098efb006d383cc47133d5ababe402fff2faa056ba20f8931a32a2a37f"} Nov 02 19:51:43.352026 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:43.351997 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7svjs" event={"ID":"a3823ef4-9aae-417c-bc9a-a04253589bf5","Type":"ContainerStarted","Data":"92d2b3f3f0e077cbd8854ef006b98009e3e9b1063e1f5e953b7b49e9f5c3df0b"} Nov 02 19:51:43.354506 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:43.354410 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" event={"ID":"d10bb7fa-2eb9-4215-b467-c2165c4c76f5","Type":"ContainerStarted","Data":"d45f957ea75ff8d971cb5a5750dd6b91d283fa455cfaafe6d4e62b41219cbcf7"} Nov 02 19:51:43.356611 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:43.356588 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z2vn7" event={"ID":"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57","Type":"ContainerStarted","Data":"e6cac76090e2631867986329c79ddc308f8d8ed1c3bb2a189929894d7e60054d"} Nov 02 19:51:43.463344 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:43.463289 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9rglx" podStartSLOduration=2.430441859 podStartE2EDuration="11.46327478s" totalImagesPullingTime="9.032832921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:43.462433045 +0000 UTC m=+12.691401002" watchObservedRunningTime="2026-11-02 19:51:43.46327478 +0000 UTC m=+12.692242733" Nov 02 19:51:44.113434 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:44.113405 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:44.351167 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:44.351128 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:44.362008 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:44.361983 2575 generic.go:356] "Generic (PLEG): container finished" podID="a3823ef4-9aae-417c-bc9a-a04253589bf5" containerID="92d2b3f3f0e077cbd8854ef006b98009e3e9b1063e1f5e953b7b49e9f5c3df0b" exitCode=0 Nov 02 19:51:44.362166 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:44.362057 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7svjs" event={"ID":"a3823ef4-9aae-417c-bc9a-a04253589bf5","Type":"ContainerDied","Data":"92d2b3f3f0e077cbd8854ef006b98009e3e9b1063e1f5e953b7b49e9f5c3df0b"} Nov 02 19:51:44.363641 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:44.363604 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nqhf8" event={"ID":"c10c2efe-3e34-4ca7-b972-680cde439609","Type":"ContainerStarted","Data":"729a43e1cdd65051f5f73b049a0699257f03b901912304aa0af5014791ce3f59"} Nov 02 19:51:44.366560 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:44.366491 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" event={"ID":"d10bb7fa-2eb9-4215-b467-c2165c4c76f5","Type":"ContainerStarted","Data":"42d66df5a4b5fb87603a812fe425c6682d319a621d361f3119fe1ebe708004c3"} Nov 02 19:51:44.477440 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:44.477333 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nqhf8" podStartSLOduration=3.680996418 podStartE2EDuration="12.47731874s" totalImagesPullingTime="8.796322322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:44.476941413 +0000 UTC m=+13.705909363" watchObservedRunningTime="2026-11-02 19:51:44.47731874 +0000 UTC m=+13.706286700" Nov 02 19:51:44.477798 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:44.477761 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-z2vn7" podStartSLOduration=3.521663721 podStartE2EDuration="12.477748168s" totalImagesPullingTime="8.956084447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:43.698910204 +0000 UTC m=+12.927878162" watchObservedRunningTime="2026-11-02 19:51:44.477748168 +0000 UTC m=+13.706716126" Nov 02 19:51:44.589230 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:44.589176 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" podStartSLOduration=2.515948988 podStartE2EDuration="12.58915938s" totalImagesPullingTime="10.073210392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:44.58873296 +0000 UTC m=+13.817700914" watchObservedRunningTime="2026-11-02 19:51:44.58915938 +0000 UTC m=+13.818127332" Nov 02 19:51:44.596893 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:44.596849 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:45.256857 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:45.256824 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:45.256857 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:45.256860 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:45.257109 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:45.256824 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:45.257109 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:45.256971 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:45.257109 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:45.257095 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:45.257260 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:45.257185 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:45.371290 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:45.371257 2575 generic.go:356] "Generic (PLEG): container finished" podID="a3823ef4-9aae-417c-bc9a-a04253589bf5" containerID="90d19e77e182fac56eb0fac99f45202dc496aeba03742adfa1d0eab74bdb634c" exitCode=0 Nov 02 19:51:45.371973 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:45.371370 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7svjs" event={"ID":"a3823ef4-9aae-417c-bc9a-a04253589bf5","Type":"ContainerDied","Data":"90d19e77e182fac56eb0fac99f45202dc496aeba03742adfa1d0eab74bdb634c"} Nov 02 19:51:45.373120 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:45.373075 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" event={"ID":"f0ed3429-8e54-4f88-b820-751eacadd01a","Type":"ContainerStarted","Data":"d088d084559b3f95e19285c358cef818a283527c8f877a736c2580ddbdcd9ba0"} Nov 02 19:51:45.585476 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:45.585377 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" podStartSLOduration=2.2969967000000002 podStartE2EDuration="13.5853576s" totalImagesPullingTime="11.2883609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:45.584688237 +0000 UTC m=+14.813656190" watchObservedRunningTime="2026-11-02 19:51:45.5853576 +0000 UTC m=+14.814325554" Nov 02 19:51:47.256516 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:47.256265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:47.257000 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:47.256265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:47.257000 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:47.256636 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:47.257000 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:47.256325 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:47.257000 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:47.256714 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:47.257000 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:47.256761 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:47.370284 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:47.370247 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:47.379644 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:47.379611 2575 generic.go:356] "Generic (PLEG): container finished" podID="a3823ef4-9aae-417c-bc9a-a04253589bf5" containerID="b5a2a7623eb0f792f49497ac05e8b5df4a280676b9945a463842b9a50ab6a586" exitCode=0 Nov 02 19:51:47.379805 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:47.379671 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7svjs" event={"ID":"a3823ef4-9aae-417c-bc9a-a04253589bf5","Type":"ContainerDied","Data":"b5a2a7623eb0f792f49497ac05e8b5df4a280676b9945a463842b9a50ab6a586"} Nov 02 19:51:47.604978 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:47.604901 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:47.776219 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:47.776181 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:47.955189 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:47.955152 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:48.617980 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:48.617946 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:48.801188 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:48.801153 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:48.977723 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:48.977695 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:48.985796 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:48.985774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:48.985868 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:48.985819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:48.985940 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:48.985928 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:48.985983 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:48.985975 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs podName:0d60e019-846f-497e-a1fb-1c972295523b nodeName:}" failed. No retries permitted until 2026-11-02 19:52:04.985962351 +0000 UTC m=+34.214930294 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs") pod "network-metrics-daemon-r67nn" (UID: "0d60e019-846f-497e-a1fb-1c972295523b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:48.986028 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:48.985928 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:48.986074 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:48.986064 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret podName:6dde0e70-73a7-4198-831e-875401c4c3cd nodeName:}" failed. No retries permitted until 2026-11-02 19:52:04.986049942 +0000 UTC m=+34.215017889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret") pod "global-pull-secret-syncer-4bpmh" (UID: "6dde0e70-73a7-4198-831e-875401c4c3cd") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:49.187188 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:49.187153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwzr\" (UniqueName: \"kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr\") pod \"network-check-target-m2tvk\" (UID: \"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9\") " pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:49.187368 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:49.187315 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:49.187368 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:49.187338 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:49.187368 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:49.187351 2575 projected.go:196] Error preparing data for projected volume kube-api-access-gnwzr for pod openshift-network-diagnostics/network-check-target-m2tvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:49.187514 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:49.187420 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr podName:b2ff0bc2-a4cf-45f3-b870-2526d024b6c9 nodeName:}" failed. No retries permitted until 2026-11-02 19:52:05.187398545 +0000 UTC m=+34.416366495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnwzr" (UniqueName: "kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr") pod "network-check-target-m2tvk" (UID: "b2ff0bc2-a4cf-45f3-b870-2526d024b6c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:49.257270 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:49.257192 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:49.257443 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:49.257328 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:49.257443 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:49.257204 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:49.257443 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:49.257410 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:49.257443 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:49.257192 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:49.257653 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:49.257496 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:49.387392 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:49.387367 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" event={"ID":"98798482-f8b5-401e-9aa2-363cda917fba","Type":"ContainerStarted","Data":"00bef1df441e520c4583dd359fd30d9c5437875f8ef2d476110463d8cb07f9af"} Nov 02 19:51:50.166330 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:50.166148 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:50.360845 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:50.360810 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:50.391756 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:50.391723 2575 generic.go:356] "Generic (PLEG): container finished" podID="a3823ef4-9aae-417c-bc9a-a04253589bf5" containerID="cf1edd4a5e93def852d6e5c0dac6521d4d44e1b1a5a6961524a8cdcaafc6fa9b" exitCode=0 Nov 02 19:51:50.391998 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:50.391820 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7svjs" event={"ID":"a3823ef4-9aae-417c-bc9a-a04253589bf5","Type":"ContainerDied","Data":"cf1edd4a5e93def852d6e5c0dac6521d4d44e1b1a5a6961524a8cdcaafc6fa9b"} Nov 02 19:51:50.397757 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:50.397720 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" event={"ID":"98798482-f8b5-401e-9aa2-363cda917fba","Type":"ContainerStarted","Data":"5352810dc82df890a78848f3af23acda94c10805777b18b29f56d9e793749a9c"} Nov 02 19:51:50.397886 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:50.397858 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" event={"ID":"98798482-f8b5-401e-9aa2-363cda917fba","Type":"ContainerStarted","Data":"197e4d3153080b225c9966a4c747a79fcea27cbadb04290a49208f22eb33dbf6"} Nov 02 19:51:50.397946 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:50.397931 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" event={"ID":"98798482-f8b5-401e-9aa2-363cda917fba","Type":"ContainerStarted","Data":"cb2677846793cf384cf02a282d4428feabdb5cfc8a69d8087fd9a6848e1debb1"} Nov 02 19:51:50.398001 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:50.397987 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" event={"ID":"98798482-f8b5-401e-9aa2-363cda917fba","Type":"ContainerStarted","Data":"d5ea17b16b8345f013fd78860ea9668bdbbd2f4fb68f2948d3eaa7beacf9dc31"} Nov 02 19:51:50.398050 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:50.398039 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" event={"ID":"98798482-f8b5-401e-9aa2-363cda917fba","Type":"ContainerStarted","Data":"da0b58e3ef8b084d590e09ef905103af3abd500af75e106a69d7d89cc249867a"} Nov 02 19:51:50.727234 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:50.727146 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:50.901021 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:50.900991 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:51.256805 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:51.256769 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:51.257287 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:51.256929 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:51.257287 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:51.256952 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:51.257287 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:51.256988 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:51.257287 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:51.257033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:51.257287 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:51.257113 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:51.402968 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:51.402935 2575 generic.go:356] "Generic (PLEG): container finished" podID="a3823ef4-9aae-417c-bc9a-a04253589bf5" containerID="2138975dc1e7fc6a2dddf3dab1f72a75c315b2cac5c251f59294947da4a9938f" exitCode=0 Nov 02 19:51:51.403134 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:51.402987 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7svjs" event={"ID":"a3823ef4-9aae-417c-bc9a-a04253589bf5","Type":"ContainerDied","Data":"2138975dc1e7fc6a2dddf3dab1f72a75c315b2cac5c251f59294947da4a9938f"} Nov 02 19:51:51.595550 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:51.595475 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9rglx_e588ae83-565a-4711-a7b7-d77b8ae24bc8/dns-node-resolver/0.log" Nov 02 19:51:52.304986 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:52.304959 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dsct6_2c6f9fda-5459-4d5a-8405-7eab0de2cb1e/node-ca/0.log" Nov 02 19:51:52.408434 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:52.408403 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" event={"ID":"98798482-f8b5-401e-9aa2-363cda917fba","Type":"ContainerStarted","Data":"bb30f00dab5950a546ff89bc9f4b6be416e25d27753825dfb049d06bf25b987f"} Nov 02 19:51:52.411463 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:52.411439 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7svjs" event={"ID":"a3823ef4-9aae-417c-bc9a-a04253589bf5","Type":"ContainerStarted","Data":"fd7c67212ce4de4846624511e27e5f0e6d1cf37643cdd1893723d3206f6b9626"} Nov 02 19:51:52.522200 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:52.522154 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7svjs" podStartSLOduration=6.808436197 podStartE2EDuration="20.522142674s" totalImagesPullingTime="13.713706477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:52.521862761 +0000 UTC m=+21.750830713" watchObservedRunningTime="2026-11-02 19:51:52.522142674 +0000 UTC m=+21.751110645" Nov 02 19:51:53.256982 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:53.256940 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:53.257153 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:53.256998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:53.257153 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:53.256995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:53.257153 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:53.257091 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:53.257259 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:53.257228 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:53.257339 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:53.257321 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:53.748403 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:53.748373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:54.113624 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:54.113358 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:54.393495 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:54.393417 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:54.419397 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:54.419359 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" event={"ID":"98798482-f8b5-401e-9aa2-363cda917fba","Type":"ContainerStarted","Data":"997d18ced0595dd4d867569f9dcdad7e398432d27b019e263d9056b6ad1e57d2"} Nov 02 19:51:54.419608 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:54.419591 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:54.435432 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:54.435406 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:54.569005 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:54.568953 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:54.646688 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:54.646616 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:51:54.646827 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:54.646731 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:51:54.647364 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:54.647333 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:51:54.676603 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:54.676555 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" podStartSLOduration=6.947429848 podStartE2EDuration="22.676543118s" totalImagesPullingTime="15.72911327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:54.676244149 +0000 UTC m=+23.905212101" watchObservedRunningTime="2026-11-02 19:51:54.676543118 +0000 UTC m=+23.905511072" Nov 02 19:51:54.772718 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:54.772686 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:55.221757 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:55.221715 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:55.256920 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:55.256888 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:55.257091 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:55.256944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:55.257091 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:55.256956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:55.257091 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:55.257018 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:55.257215 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:55.257140 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:55.257285 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:55.257251 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:55.421655 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:55.421630 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:55.421816 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:55.421699 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:55.436322 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:55.436295 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:51:55.817950 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:55.817920 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:55.989688 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:55.989660 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:56.163092 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:56.163060 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:56.332033 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:56.331998 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:56.524032 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:56.523960 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:56.735807 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:56.735775 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:57.058952 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:57.058924 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:57.256408 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:57.256380 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:57.256544 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:57.256392 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:57.256544 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:57.256483 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:57.256544 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:57.256494 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:57.256662 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:57.256587 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:57.256696 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:57.256657 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:57.936173 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:57.936138 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:58.132373 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:58.132342 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:58.326080 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:58.326003 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:58.534032 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:58.533999 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:58.696836 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:58.696808 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:58.869235 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:58.869206 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:59.257128 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:59.257100 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:51:59.257542 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:59.257100 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:51:59.257542 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:59.257215 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:51:59.257542 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:59.257227 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:51:59.257542 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:59.257284 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:51:59.257542 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:51:59.257401 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:51:59.487762 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:59.487731 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:59.660221 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:59.660195 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:51:59.927456 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:51:59.927379 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:00.130197 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:00.130154 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:00.331674 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:00.331602 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:00.526838 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:00.526807 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:00.732962 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:00.732931 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:01.257455 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:01.257425 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:01.257625 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:01.257513 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:52:01.257625 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:01.257586 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:01.257724 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:01.257666 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:52:01.257724 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:01.257670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:01.257805 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:01.257775 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:52:01.928161 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:01.928128 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:02.124215 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:02.124183 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:02.323895 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:02.323805 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:02.528359 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:02.528329 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:03.258983 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:03.258958 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:03.259368 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:03.259029 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:03.259368 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:03.259052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:03.259368 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:03.259136 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:52:03.259368 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:03.259183 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:52:03.259368 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:03.259239 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:52:03.738053 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:03.738021 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:03.925794 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:03.925764 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:04.130054 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:04.129973 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:04.295527 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:04.295489 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:04.471352 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:04.471321 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:04.643131 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:04.643099 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:04.813425 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:04.813342 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:04.996323 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:04.996289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:04.996493 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:04.996342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:04.996493 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:04.996435 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:52:04.996571 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:04.996499 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs podName:0d60e019-846f-497e-a1fb-1c972295523b nodeName:}" failed. No retries permitted until 2026-11-02 19:52:36.996483528 +0000 UTC m=+66.225451475 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs") pod "network-metrics-daemon-r67nn" (UID: "0d60e019-846f-497e-a1fb-1c972295523b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:52:04.996571 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:04.996439 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:52:04.996571 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:04.996561 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret podName:6dde0e70-73a7-4198-831e-875401c4c3cd nodeName:}" failed. No retries permitted until 2026-11-02 19:52:36.996549701 +0000 UTC m=+66.225517632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret") pod "global-pull-secret-syncer-4bpmh" (UID: "6dde0e70-73a7-4198-831e-875401c4c3cd") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:52:05.198126 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:05.198090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwzr\" (UniqueName: \"kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr\") pod \"network-check-target-m2tvk\" (UID: \"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9\") " pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:05.198280 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:05.198208 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:52:05.198280 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:05.198221 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:52:05.198280 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:05.198230 2575 projected.go:196] Error preparing data for projected volume kube-api-access-gnwzr for pod openshift-network-diagnostics/network-check-target-m2tvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:52:05.198280 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:05.198277 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr podName:b2ff0bc2-a4cf-45f3-b870-2526d024b6c9 nodeName:}" failed. No retries permitted until 2026-11-02 19:52:37.198263116 +0000 UTC m=+66.427231047 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnwzr" (UniqueName: "kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr") pod "network-check-target-m2tvk" (UID: "b2ff0bc2-a4cf-45f3-b870-2526d024b6c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:52:05.256509 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:05.256476 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:05.256509 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:05.256501 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:05.256740 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:05.256594 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:52:05.256740 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:05.256643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:05.256740 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:05.256693 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:52:05.256740 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:05.256640 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:52:05.380883 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:05.380853 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:05.612771 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:05.612695 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:05.967863 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:05.967832 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:06.133913 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:06.133868 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:06.305225 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:06.305144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:06.475619 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:06.475588 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:06.810688 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:06.810661 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:07.256262 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:07.256235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:07.256427 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:07.256298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:07.256427 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:07.256296 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:07.256427 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:07.256381 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:52:07.256525 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:07.256426 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:52:07.256525 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:07.256497 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:52:07.581490 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:07.581411 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:07.747158 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:07.747124 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:09.256589 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:09.256556 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:09.256589 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:09.256581 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:09.257035 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:09.256683 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:52:09.257035 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:09.256752 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:52:09.257035 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:09.256799 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:09.257035 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:09.256907 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:52:10.246326 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:10.246283 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:10.445351 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:10.445321 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:10.616594 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:10.616502 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:10.857102 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:10.857070 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:11.065419 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:11.065385 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:11.257117 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:11.257087 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:11.257263 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:11.257127 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:11.257263 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:11.257174 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:52:11.257263 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:11.257223 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:52:11.257363 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:11.257277 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:11.257363 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:11.257326 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:52:12.538205 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:12.537956 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4bpmh"] Nov 02 19:52:12.538534 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:12.538304 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:12.538534 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:12.538419 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:52:12.541155 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:12.541096 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2tvk"] Nov 02 19:52:12.541260 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:12.541190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:12.541317 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:12.541281 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:52:12.541512 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:12.541488 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r67nn"] Nov 02 19:52:12.541608 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:12.541571 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:12.541669 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:12.541653 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:52:14.257020 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:14.256947 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:14.257020 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:14.257008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:14.257406 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:14.257089 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:14.257406 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:14.257102 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:52:14.257406 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:14.257150 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:52:14.257406 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:14.257232 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:52:14.564755 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:14.564645 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:14.753516 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:14.753486 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:14.933390 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:14.933356 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:15.163308 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:15.163280 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:16.256683 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.256652 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:16.257152 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.256692 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:16.257152 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.256745 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:16.257152 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:16.256764 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2tvk" podUID="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" Nov 02 19:52:16.257152 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:16.256823 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r67nn" podUID="0d60e019-846f-497e-a1fb-1c972295523b" Nov 02 19:52:16.257152 ip-10-0-134-194 kubenswrapper[2575]: E1102 19:52:16.256887 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bpmh" podUID="6dde0e70-73a7-4198-831e-875401c4c3cd" Nov 02 19:52:16.401041 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.401011 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:16.566925 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.566899 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:16.577605 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.577586 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeReady" Nov 02 19:52:16.577693 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.577688 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 19:52:16.685275 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.685245 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jqtjb"] Nov 02 19:52:16.687048 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.687028 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:16.732127 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.732100 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-92vwp"] Nov 02 19:52:16.733863 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.733848 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4b5v5"] Nov 02 19:52:16.733987 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.733974 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-92vwp" Nov 02 19:52:16.735663 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.735647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.735891 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.735862 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 19:52:16.736340 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.736319 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvmjq\"" Nov 02 19:52:16.736441 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.736426 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 19:52:16.783136 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.783113 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 19:52:16.783240 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.783139 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 19:52:16.783240 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.783149 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 19:52:16.783240 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.783140 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d5pq8\"" Nov 02 19:52:16.784277 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.784252 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 19:52:16.784277 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.784273 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 19:52:16.784393 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.784322 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 19:52:16.784393 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.784333 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7qs77\"" Nov 02 19:52:16.784393 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.784363 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 19:52:16.794708 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.794685 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jqtjb"] Nov 02 19:52:16.795312 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.795291 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-92vwp"] Nov 02 19:52:16.795907 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.795888 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4b5v5"] Nov 02 19:52:16.874407 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.874319 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-data-volume\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.874407 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.874357 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a409cedc-aedc-4130-aad6-4405f1539799-config-volume\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:16.874407 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.874394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8mg9\" (UniqueName: \"kubernetes.io/projected/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-kube-api-access-x8mg9\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.874660 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.874527 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a409cedc-aedc-4130-aad6-4405f1539799-metrics-tls\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:16.874660 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.874560 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a409cedc-aedc-4130-aad6-4405f1539799-tmp-dir\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:16.874660 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.874589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.874752 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.874675 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8ws\" (UniqueName: \"kubernetes.io/projected/a409cedc-aedc-4130-aad6-4405f1539799-kube-api-access-9x8ws\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:16.874752 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.874714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-crio-socket\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.874752 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.874739 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.874848 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.874774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ab821fc-d8d9-4ed9-86df-56979b6ac88a-cert\") pod \"ingress-canary-92vwp\" (UID: \"2ab821fc-d8d9-4ed9-86df-56979b6ac88a\") " pod="openshift-ingress-canary/ingress-canary-92vwp" Nov 02 19:52:16.874901 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.874845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwzsj\" (UniqueName: \"kubernetes.io/projected/2ab821fc-d8d9-4ed9-86df-56979b6ac88a-kube-api-access-wwzsj\") pod \"ingress-canary-92vwp\" (UID: \"2ab821fc-d8d9-4ed9-86df-56979b6ac88a\") " pod="openshift-ingress-canary/ingress-canary-92vwp" Nov 02 19:52:16.975433 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a409cedc-aedc-4130-aad6-4405f1539799-metrics-tls\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:16.975433 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a409cedc-aedc-4130-aad6-4405f1539799-tmp-dir\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:16.975696 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.975696 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8ws\" (UniqueName: \"kubernetes.io/projected/a409cedc-aedc-4130-aad6-4405f1539799-kube-api-access-9x8ws\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:16.975696 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-crio-socket\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.975846 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.975846 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975733 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-crio-socket\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.975846 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ab821fc-d8d9-4ed9-86df-56979b6ac88a-cert\") pod \"ingress-canary-92vwp\" (UID: \"2ab821fc-d8d9-4ed9-86df-56979b6ac88a\") " pod="openshift-ingress-canary/ingress-canary-92vwp" Nov 02 19:52:16.975846 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975786 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwzsj\" (UniqueName: \"kubernetes.io/projected/2ab821fc-d8d9-4ed9-86df-56979b6ac88a-kube-api-access-wwzsj\") pod \"ingress-canary-92vwp\" (UID: \"2ab821fc-d8d9-4ed9-86df-56979b6ac88a\") " pod="openshift-ingress-canary/ingress-canary-92vwp" Nov 02 19:52:16.975846 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-data-volume\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.975846 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975813 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a409cedc-aedc-4130-aad6-4405f1539799-tmp-dir\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:16.975846 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a409cedc-aedc-4130-aad6-4405f1539799-config-volume\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:16.976184 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.975913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8mg9\" (UniqueName: \"kubernetes.io/projected/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-kube-api-access-x8mg9\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.976280 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.976258 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-data-volume\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.976366 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.976350 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.976515 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.976496 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a409cedc-aedc-4130-aad6-4405f1539799-config-volume\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:16.979482 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.979452 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:16.979587 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.979488 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a409cedc-aedc-4130-aad6-4405f1539799-metrics-tls\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:16.979587 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:16.979554 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ab821fc-d8d9-4ed9-86df-56979b6ac88a-cert\") pod \"ingress-canary-92vwp\" (UID: \"2ab821fc-d8d9-4ed9-86df-56979b6ac88a\") " pod="openshift-ingress-canary/ingress-canary-92vwp" Nov 02 19:52:17.028890 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.028835 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8mg9\" (UniqueName: \"kubernetes.io/projected/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-kube-api-access-x8mg9\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:17.029077 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.029056 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8ws\" (UniqueName: \"kubernetes.io/projected/a409cedc-aedc-4130-aad6-4405f1539799-kube-api-access-9x8ws\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:17.029139 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.029112 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwzsj\" (UniqueName: \"kubernetes.io/projected/2ab821fc-d8d9-4ed9-86df-56979b6ac88a-kube-api-access-wwzsj\") pod \"ingress-canary-92vwp\" (UID: \"2ab821fc-d8d9-4ed9-86df-56979b6ac88a\") " pod="openshift-ingress-canary/ingress-canary-92vwp" Nov 02 19:52:17.042764 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.042741 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8478dfbff8-ldrtw"] Nov 02 19:52:17.045059 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.045041 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.045582 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.045557 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-92vwp" Nov 02 19:52:17.052013 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.051997 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:17.094051 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.093977 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 19:52:17.094235 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.094154 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5zj4s\"" Nov 02 19:52:17.094299 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.094244 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 19:52:17.094299 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.094259 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 19:52:17.094299 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.094265 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 19:52:17.094440 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.094314 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 19:52:17.094440 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.094331 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 19:52:17.094440 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.094416 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 19:52:17.100163 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.100137 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8478dfbff8-ldrtw"] Nov 02 19:52:17.177134 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.177102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-oauth-serving-cert\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.177134 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.177134 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-console-config\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.177314 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.177150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-service-ca\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.177314 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.177265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-oauth-config\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.177314 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.177297 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafdd19a-1c68-4429-9519-7da446447921-tmp\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.177407 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.177313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmns\" (UniqueName: \"kubernetes.io/projected/fafdd19a-1c68-4429-9519-7da446447921-kube-api-access-6bmns\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.177436 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.177405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-serving-cert\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.277840 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.277655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafdd19a-1c68-4429-9519-7da446447921-tmp\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.278738 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.277849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmns\" (UniqueName: \"kubernetes.io/projected/fafdd19a-1c68-4429-9519-7da446447921-kube-api-access-6bmns\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.278738 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.277927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-serving-cert\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.278738 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.277953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-oauth-serving-cert\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.278738 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.277978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-console-config\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.278738 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.278003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-service-ca\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.278738 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.278064 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafdd19a-1c68-4429-9519-7da446447921-tmp\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.278738 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.278068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-oauth-config\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.278738 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.278665 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-oauth-serving-cert\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.278738 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.278717 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-service-ca\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.278738 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.278728 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-console-config\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.280545 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.280525 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-oauth-config\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.280623 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.280614 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-serving-cert\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.297648 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.297628 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:17.330747 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.330720 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmns\" (UniqueName: \"kubernetes.io/projected/fafdd19a-1c68-4429-9519-7da446447921-kube-api-access-6bmns\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.361194 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.361168 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:17.433970 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.433433 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4b5v5"] Nov 02 19:52:17.470428 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.470404 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4b5v5" event={"ID":"ef721fd3-36cc-4fc3-b17c-6ae016daf10c","Type":"ContainerStarted","Data":"ad4140c0ae0093f04b2e4d9ec3d2214f289e7f173039b984103ae84a8a89c560"} Nov 02 19:52:17.478441 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.478416 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-92vwp"] Nov 02 19:52:17.488899 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:17.488852 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab821fc_d8d9_4ed9_86df_56979b6ac88a.slice/crio-9957bdb38b1f268e3b8c2ca16ad13609c65ae293836d9f9923f4c5a1e1e0826e WatchSource:0}: Error finding container 9957bdb38b1f268e3b8c2ca16ad13609c65ae293836d9f9923f4c5a1e1e0826e: Status 404 returned error can't find the container with id 9957bdb38b1f268e3b8c2ca16ad13609c65ae293836d9f9923f4c5a1e1e0826e Nov 02 19:52:17.660286 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.660251 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jqtjb"] Nov 02 19:52:17.668432 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:17.668394 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda409cedc_aedc_4130_aad6_4405f1539799.slice/crio-1dde184d4c22da9ef2897be99a6e57bd6448f62f35cad908c9e83dadb882b4eb WatchSource:0}: Error finding container 1dde184d4c22da9ef2897be99a6e57bd6448f62f35cad908c9e83dadb882b4eb: Status 404 returned error can't find the container with id 1dde184d4c22da9ef2897be99a6e57bd6448f62f35cad908c9e83dadb882b4eb Nov 02 19:52:17.746412 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:17.746339 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8478dfbff8-ldrtw"] Nov 02 19:52:17.754857 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:17.754832 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafdd19a_1c68_4429_9519_7da446447921.slice/crio-263cb0b0be40e4059ac1a4f5018e966d5ee3bfd6a417af4702b2846d7ed335d8 WatchSource:0}: Error finding container 263cb0b0be40e4059ac1a4f5018e966d5ee3bfd6a417af4702b2846d7ed335d8: Status 404 returned error can't find the container with id 263cb0b0be40e4059ac1a4f5018e966d5ee3bfd6a417af4702b2846d7ed335d8 Nov 02 19:52:18.257066 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.257031 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:18.257248 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.257030 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:18.257248 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.257161 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:18.306158 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.306064 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:52:18.306158 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.306140 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:52:18.306639 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.306232 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gf4xf\"" Nov 02 19:52:18.306639 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.306246 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sk47\"" Nov 02 19:52:18.352783 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.352750 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:52:18.352981 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.352947 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:52:18.352981 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.352943 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ltmn5\"" Nov 02 19:52:18.476519 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.476483 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4b5v5" event={"ID":"ef721fd3-36cc-4fc3-b17c-6ae016daf10c","Type":"ContainerStarted","Data":"69880d42b8caa8a5cf49edaad83fd90ef61cd27210a85c5a4be117e1f1eec317"} Nov 02 19:52:18.478566 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.478542 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8478dfbff8-ldrtw" event={"ID":"fafdd19a-1c68-4429-9519-7da446447921","Type":"ContainerStarted","Data":"263cb0b0be40e4059ac1a4f5018e966d5ee3bfd6a417af4702b2846d7ed335d8"} Nov 02 19:52:18.479832 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.479806 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jqtjb" event={"ID":"a409cedc-aedc-4130-aad6-4405f1539799","Type":"ContainerStarted","Data":"1dde184d4c22da9ef2897be99a6e57bd6448f62f35cad908c9e83dadb882b4eb"} Nov 02 19:52:18.480801 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:18.480782 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-92vwp" event={"ID":"2ab821fc-d8d9-4ed9-86df-56979b6ac88a","Type":"ContainerStarted","Data":"9957bdb38b1f268e3b8c2ca16ad13609c65ae293836d9f9923f4c5a1e1e0826e"} Nov 02 19:52:19.064102 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:19.063972 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:19.230480 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:19.230444 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:19.487267 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:19.487232 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4b5v5" event={"ID":"ef721fd3-36cc-4fc3-b17c-6ae016daf10c","Type":"ContainerStarted","Data":"16f9bb9a059a012a16c071172074e5310a499ee99a1bbbff188e75200c6902d2"} Nov 02 19:52:20.410307 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:20.410234 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:21.064998 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.064962 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:21.303563 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.303344 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:21.496804 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.496769 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4b5v5" event={"ID":"ef721fd3-36cc-4fc3-b17c-6ae016daf10c","Type":"ContainerStarted","Data":"2df70a109e50d68ff862ab50725bf2e10ba5802877d3904d228d4d578344b02d"} Nov 02 19:52:21.498847 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.498793 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jqtjb" event={"ID":"a409cedc-aedc-4130-aad6-4405f1539799","Type":"ContainerStarted","Data":"5469521324b3e290eb4e965c3c5193abf7466166debb907f919fa4f875a3df9a"} Nov 02 19:52:21.498847 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.498848 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:21.499026 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.498894 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jqtjb" event={"ID":"a409cedc-aedc-4130-aad6-4405f1539799","Type":"ContainerStarted","Data":"a0fc53995519601f123463be2aa203646346545e0c1a56cdedaf8894222f9f39"} Nov 02 19:52:21.500524 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.500497 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-92vwp" event={"ID":"2ab821fc-d8d9-4ed9-86df-56979b6ac88a","Type":"ContainerStarted","Data":"cef7fda9d0a2df267a34fb416709183bca7483c97bd31b2b17c9fe1299825f9d"} Nov 02 19:52:21.582285 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.582209 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:21.634670 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.634626 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4b5v5" podStartSLOduration=2.285854398 podStartE2EDuration="5.634612731s" totalImagesPullingTime="3.348758333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:52:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:21.634091097 +0000 UTC m=+50.863059051" watchObservedRunningTime="2026-11-02 19:52:21.634612731 +0000 UTC m=+50.863580680" Nov 02 19:52:21.741558 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.741508 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-jqtjb" podStartSLOduration=2.485932007 podStartE2EDuration="5.741493416s" totalImagesPullingTime="3.255561409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:21.741010958 +0000 UTC m=+50.969978911" watchObservedRunningTime="2026-11-02 19:52:21.741493416 +0000 UTC m=+50.970461365" Nov 02 19:52:21.744542 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.744521 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5"] Nov 02 19:52:21.746385 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.746368 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:21.771598 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.771573 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:21.795167 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.795149 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 19:52:21.795308 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.795229 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 19:52:21.795308 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.795237 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 19:52:21.795417 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.795319 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 19:52:21.795417 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.795327 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-t8rrc\"" Nov 02 19:52:21.795417 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.795342 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 19:52:21.828701 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.828682 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5"] Nov 02 19:52:21.849522 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.849454 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-92vwp" podStartSLOduration=2.019360595 podStartE2EDuration="5.849443162s" totalImagesPullingTime="3.830082567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:21.849224876 +0000 UTC m=+51.078192828" watchObservedRunningTime="2026-11-02 19:52:21.849443162 +0000 UTC m=+51.078411114" Nov 02 19:52:21.868902 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.868866 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-287j2"] Nov 02 19:52:21.870811 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.870794 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:21.872098 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.872077 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9"] Nov 02 19:52:21.873762 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.873743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:21.917921 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.917896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0170aa95-6bc5-4107-98f6-7beea4227f4f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:21.918069 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.917952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0170aa95-6bc5-4107-98f6-7beea4227f4f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:21.918069 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.917971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0170aa95-6bc5-4107-98f6-7beea4227f4f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:21.918069 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.918054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t966\" (UniqueName: \"kubernetes.io/projected/0170aa95-6bc5-4107-98f6-7beea4227f4f-kube-api-access-9t966\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:21.919938 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.919920 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 19:52:21.952466 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.952440 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9"] Nov 02 19:52:21.957914 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:21.957856 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:22.012012 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.011984 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 19:52:22.012177 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.012105 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jbw4z\"" Nov 02 19:52:22.012177 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.012105 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 19:52:22.012177 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.012109 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 19:52:22.012283 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.012196 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 19:52:22.012283 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.012213 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xjzch\"" Nov 02 19:52:22.012283 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.012274 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 19:52:22.018566 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.018548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/092fd489-4d42-46d1-a178-e95204a400e8-metrics-client-ca\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.018611 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.018573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0170aa95-6bc5-4107-98f6-7beea4227f4f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:22.018611 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.018591 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-tls\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.018689 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.018610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t966\" (UniqueName: \"kubernetes.io/projected/0170aa95-6bc5-4107-98f6-7beea4227f4f-kube-api-access-9t966\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:22.018689 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.018638 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmg6s\" (UniqueName: \"kubernetes.io/projected/092fd489-4d42-46d1-a178-e95204a400e8-kube-api-access-jmg6s\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.018756 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.018707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.018756 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.018729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.018836 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.018758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0170aa95-6bc5-4107-98f6-7beea4227f4f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:22.018836 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.018830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0170aa95-6bc5-4107-98f6-7beea4227f4f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:22.019001 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.018984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-sys\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.019068 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.019030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-wtmp\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.019144 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.019121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.019199 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.019164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.019321 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.019297 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.019321 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.019303 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0170aa95-6bc5-4107-98f6-7beea4227f4f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:22.019415 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.019352 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbxrc\" (UniqueName: \"kubernetes.io/projected/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-api-access-xbxrc\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.019415 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.019400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.019503 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.019418 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.019503 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.019443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-root\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.019589 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.019509 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-textfile\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.021835 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.021816 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0170aa95-6bc5-4107-98f6-7beea4227f4f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:22.021927 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.021866 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0170aa95-6bc5-4107-98f6-7beea4227f4f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:22.071348 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.071324 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t966\" (UniqueName: \"kubernetes.io/projected/0170aa95-6bc5-4107-98f6-7beea4227f4f-kube-api-access-9t966\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:22.120346 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120313 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-sys\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.120448 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120350 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-wtmp\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.120448 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.120448 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120436 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-sys\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.120580 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120500 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-wtmp\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.120580 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.120660 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.120660 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbxrc\" (UniqueName: \"kubernetes.io/projected/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-api-access-xbxrc\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.120660 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.120660 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.120834 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-root\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.120834 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120700 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-textfile\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.120834 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/092fd489-4d42-46d1-a178-e95204a400e8-metrics-client-ca\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.120834 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-tls\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.120834 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120774 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-root\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.120834 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120771 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.120834 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmg6s\" (UniqueName: \"kubernetes.io/projected/092fd489-4d42-46d1-a178-e95204a400e8-kube-api-access-jmg6s\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.121263 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.121263 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.120869 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.121476 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.121442 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.121549 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.121491 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/092fd489-4d42-46d1-a178-e95204a400e8-metrics-client-ca\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.121549 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.121496 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.121655 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.121637 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-textfile\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.121729 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.121702 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.123139 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.123110 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-tls\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.123242 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.123188 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.123242 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.123200 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.123473 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.123455 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.127983 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.127959 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:22.174126 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.174101 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmg6s\" (UniqueName: \"kubernetes.io/projected/092fd489-4d42-46d1-a178-e95204a400e8-kube-api-access-jmg6s\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.177716 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.174775 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbxrc\" (UniqueName: \"kubernetes.io/projected/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-api-access-xbxrc\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.184537 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.184520 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:22.189200 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.189187 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:22.193912 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:22.193888 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092fd489_4d42_46d1_a178_e95204a400e8.slice/crio-58558a20b85541cdf8b2dbde0ee288393a098121a06a7d15ad31d4ad86139082 WatchSource:0}: Error finding container 58558a20b85541cdf8b2dbde0ee288393a098121a06a7d15ad31d4ad86139082: Status 404 returned error can't find the container with id 58558a20b85541cdf8b2dbde0ee288393a098121a06a7d15ad31d4ad86139082 Nov 02 19:52:22.293935 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.293866 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:22.358012 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.357963 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:22.505970 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.505934 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8478dfbff8-ldrtw" event={"ID":"fafdd19a-1c68-4429-9519-7da446447921","Type":"ContainerStarted","Data":"519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7"} Nov 02 19:52:22.507071 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.507048 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-287j2" event={"ID":"092fd489-4d42-46d1-a178-e95204a400e8","Type":"ContainerStarted","Data":"58558a20b85541cdf8b2dbde0ee288393a098121a06a7d15ad31d4ad86139082"} Nov 02 19:52:22.553134 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.553098 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:22.605393 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.605365 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9"] Nov 02 19:52:22.612456 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.612413 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8478dfbff8-ldrtw" podStartSLOduration=1.754173735 podStartE2EDuration="5.612401161s" totalImagesPullingTime="3.858227426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:22.611744221 +0000 UTC m=+51.840712174" watchObservedRunningTime="2026-11-02 19:52:22.612401161 +0000 UTC m=+51.841369114" Nov 02 19:52:22.614128 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:22.614104 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d86ef2_bb3e_4128_a81e_875fd9f6c527.slice/crio-6ecaa60df66aea858c5e29edde6204dd9e9ac319b453223e0e3d2100da7b0294 WatchSource:0}: Error finding container 6ecaa60df66aea858c5e29edde6204dd9e9ac319b453223e0e3d2100da7b0294: Status 404 returned error can't find the container with id 6ecaa60df66aea858c5e29edde6204dd9e9ac319b453223e0e3d2100da7b0294 Nov 02 19:52:22.720211 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:22.720176 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5"] Nov 02 19:52:22.728708 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:22.728680 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0170aa95_6bc5_4107_98f6_7beea4227f4f.slice/crio-2f28cfd977351c65133951958224c2e26bfcd8673d78cb02f2e20d41e0c6b978 WatchSource:0}: Error finding container 2f28cfd977351c65133951958224c2e26bfcd8673d78cb02f2e20d41e0c6b978: Status 404 returned error can't find the container with id 2f28cfd977351c65133951958224c2e26bfcd8673d78cb02f2e20d41e0c6b978 Nov 02 19:52:23.098787 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:23.098759 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:23.302012 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:23.301958 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:23.511420 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:23.511382 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" event={"ID":"c5d86ef2-bb3e-4128-a81e-875fd9f6c527","Type":"ContainerStarted","Data":"6ecaa60df66aea858c5e29edde6204dd9e9ac319b453223e0e3d2100da7b0294"} Nov 02 19:52:23.513533 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:23.513507 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" event={"ID":"0170aa95-6bc5-4107-98f6-7beea4227f4f","Type":"ContainerStarted","Data":"0a55b9ad44315aa0028dbe3f4f4cc7596e38bd2661746c0ff3b646f82cee6317"} Nov 02 19:52:23.513635 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:23.513565 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" event={"ID":"0170aa95-6bc5-4107-98f6-7beea4227f4f","Type":"ContainerStarted","Data":"0715610b5c8fe3a98a46c338b44d8dafaaab8e6277a1c21678c32cb47b486b14"} Nov 02 19:52:23.513635 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:23.513628 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" event={"ID":"0170aa95-6bc5-4107-98f6-7beea4227f4f","Type":"ContainerStarted","Data":"2f28cfd977351c65133951958224c2e26bfcd8673d78cb02f2e20d41e0c6b978"} Nov 02 19:52:23.944005 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:23.943966 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:24.175904 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.175857 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:24.436200 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.436175 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7868584944-ktpmj"] Nov 02 19:52:24.438778 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.438759 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.449917 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.449893 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7868584944-ktpmj"] Nov 02 19:52:24.496225 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.494831 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 19:52:24.496225 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.494918 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 19:52:24.496225 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.494960 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-t224c\"" Nov 02 19:52:24.496225 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.494987 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 19:52:24.496225 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.495037 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 19:52:24.496225 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.495075 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 19:52:24.496225 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.495089 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-barv9dd6r2s2h\"" Nov 02 19:52:24.519595 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.519500 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" event={"ID":"c5d86ef2-bb3e-4128-a81e-875fd9f6c527","Type":"ContainerStarted","Data":"477f14b574f900003f8a808a185c31caea0e511ef653ba0680a25391c0899814"} Nov 02 19:52:24.521199 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.521175 2575 generic.go:356] "Generic (PLEG): container finished" podID="092fd489-4d42-46d1-a178-e95204a400e8" containerID="25fc319639bab50691dea2ed127def0d71c890e4cb32555b9e014181079b1da8" exitCode=0 Nov 02 19:52:24.521295 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.521258 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-287j2" event={"ID":"092fd489-4d42-46d1-a178-e95204a400e8","Type":"ContainerDied","Data":"25fc319639bab50691dea2ed127def0d71c890e4cb32555b9e014181079b1da8"} Nov 02 19:52:24.523464 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.523446 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" event={"ID":"0170aa95-6bc5-4107-98f6-7beea4227f4f","Type":"ContainerStarted","Data":"17160a67bbae0b7db5b5cfd67d68d38013dca8f7d9d043bcebdb4bee83f88c51"} Nov 02 19:52:24.538061 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.538037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-grpc-tls\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.538239 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.538213 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.538506 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.538485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vknpr\" (UniqueName: \"kubernetes.io/projected/367d4910-a962-43b9-a7e6-a58b47636a24-kube-api-access-vknpr\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.538582 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.538540 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367d4910-a962-43b9-a7e6-a58b47636a24-tmp\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.538645 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.538583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/367d4910-a962-43b9-a7e6-a58b47636a24-metrics-client-ca\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.538645 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.538621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.538751 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.538690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.538751 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.538726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.538847 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.538756 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-tls\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.639631 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.639579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.639824 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.639634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vknpr\" (UniqueName: \"kubernetes.io/projected/367d4910-a962-43b9-a7e6-a58b47636a24-kube-api-access-vknpr\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.639824 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.639671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367d4910-a962-43b9-a7e6-a58b47636a24-tmp\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.639824 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.639704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/367d4910-a962-43b9-a7e6-a58b47636a24-metrics-client-ca\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.639824 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.639747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.639824 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.639804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.640286 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.639830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.640286 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.639862 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-tls\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.640286 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.639907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-grpc-tls\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.642009 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.641627 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367d4910-a962-43b9-a7e6-a58b47636a24-tmp\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.643110 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.643057 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/367d4910-a962-43b9-a7e6-a58b47636a24-metrics-client-ca\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.644111 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.644087 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.644194 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.644175 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-grpc-tls\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.644636 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.644615 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.644765 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.644694 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.645407 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.645174 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-tls\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.645533 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.645513 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.697182 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.697112 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vknpr\" (UniqueName: \"kubernetes.io/projected/367d4910-a962-43b9-a7e6-a58b47636a24-kube-api-access-vknpr\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.765918 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.765848 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:24.767717 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:24.767671 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" podStartSLOduration=2.190616126 podStartE2EDuration="3.767658385s" totalImagesPullingTime="1.577042259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:24.660277908 +0000 UTC m=+53.889245862" watchObservedRunningTime="2026-11-02 19:52:24.767658385 +0000 UTC m=+53.996626337" Nov 02 19:52:25.240641 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:25.240606 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7868584944-ktpmj"] Nov 02 19:52:25.248941 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:25.248915 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367d4910_a962_43b9_a7e6_a58b47636a24.slice/crio-99f0054bcd118b2b376e5e18f7804638eb0cf21297f498e7c21d07305461efb7 WatchSource:0}: Error finding container 99f0054bcd118b2b376e5e18f7804638eb0cf21297f498e7c21d07305461efb7: Status 404 returned error can't find the container with id 99f0054bcd118b2b376e5e18f7804638eb0cf21297f498e7c21d07305461efb7 Nov 02 19:52:25.528858 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:25.528768 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-287j2" event={"ID":"092fd489-4d42-46d1-a178-e95204a400e8","Type":"ContainerStarted","Data":"b3ad9750dd9fe79320429e5310ce5c935b765091205cceeba3178b2e5ab56bf2"} Nov 02 19:52:25.528858 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:25.528822 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-287j2" event={"ID":"092fd489-4d42-46d1-a178-e95204a400e8","Type":"ContainerStarted","Data":"aeecaf217999e5f41bf52e08fcbf318059ff82498a1a902864143bc0b06679eb"} Nov 02 19:52:25.530005 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:25.529981 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" event={"ID":"367d4910-a962-43b9-a7e6-a58b47636a24","Type":"ContainerStarted","Data":"99f0054bcd118b2b376e5e18f7804638eb0cf21297f498e7c21d07305461efb7"} Nov 02 19:52:25.531962 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:25.531938 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" event={"ID":"c5d86ef2-bb3e-4128-a81e-875fd9f6c527","Type":"ContainerStarted","Data":"8e6f1e3ee888665017c80393388ef7d4410e7351b80689f90a34a42f954b7e0e"} Nov 02 19:52:25.532042 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:25.531982 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" event={"ID":"c5d86ef2-bb3e-4128-a81e-875fd9f6c527","Type":"ContainerStarted","Data":"6caa13b4e18d4454e2062531b93c3ce0ab6927d01350d5a9452b8419fd0d4753"} Nov 02 19:52:25.636764 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:25.636713 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-287j2" podStartSLOduration=3.074398249 podStartE2EDuration="4.636699703s" totalImagesPullingTime="1.482677169s" totalInitContainerRuntime="79.624285ms" isStatefulPod=true podCreationTimestamp="2026-11-02 19:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:25.63546038 +0000 UTC m=+54.864428357" watchObservedRunningTime="2026-11-02 19:52:25.636699703 +0000 UTC m=+54.865667667" Nov 02 19:52:25.745849 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:25.744434 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" podStartSLOduration=2.970706232 podStartE2EDuration="4.744413726s" totalImagesPullingTime="1.773707494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:25.742718129 +0000 UTC m=+54.971686084" watchObservedRunningTime="2026-11-02 19:52:25.744413726 +0000 UTC m=+54.973381680" Nov 02 19:52:25.918764 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:25.918732 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:26.103260 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.103227 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:26.270764 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.270688 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:26.439596 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.439564 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:26.490550 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.490520 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5"] Nov 02 19:52:26.492672 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.492653 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:26.545443 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.545365 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5"] Nov 02 19:52:26.555055 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.554925 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/002e660f-77e7-4384-a7ca-514cc40a4cb6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6qrl5\" (UID: \"002e660f-77e7-4384-a7ca-514cc40a4cb6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:26.555055 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.554977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw92p\" (UniqueName: \"kubernetes.io/projected/002e660f-77e7-4384-a7ca-514cc40a4cb6-kube-api-access-mw92p\") pod \"monitoring-plugin-77f4f76594-6qrl5\" (UID: \"002e660f-77e7-4384-a7ca-514cc40a4cb6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:26.608462 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.608426 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:26.634061 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.634042 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vg4mb\"" Nov 02 19:52:26.634140 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.634042 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 19:52:26.656511 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.656482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/002e660f-77e7-4384-a7ca-514cc40a4cb6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6qrl5\" (UID: \"002e660f-77e7-4384-a7ca-514cc40a4cb6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:26.656511 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.656513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw92p\" (UniqueName: \"kubernetes.io/projected/002e660f-77e7-4384-a7ca-514cc40a4cb6-kube-api-access-mw92p\") pod \"monitoring-plugin-77f4f76594-6qrl5\" (UID: \"002e660f-77e7-4384-a7ca-514cc40a4cb6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:26.658897 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.658867 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/002e660f-77e7-4384-a7ca-514cc40a4cb6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6qrl5\" (UID: \"002e660f-77e7-4384-a7ca-514cc40a4cb6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:26.709529 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.709499 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw92p\" (UniqueName: \"kubernetes.io/projected/002e660f-77e7-4384-a7ca-514cc40a4cb6-kube-api-access-mw92p\") pod \"monitoring-plugin-77f4f76594-6qrl5\" (UID: \"002e660f-77e7-4384-a7ca-514cc40a4cb6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:26.789191 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.789161 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-796b7c8599-d2vtr"] Nov 02 19:52:26.792261 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.792241 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.804134 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.804117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:26.842090 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.842060 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 19:52:26.842249 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.842116 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 19:52:26.842249 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.842165 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-94vpabn2q7b5i\"" Nov 02 19:52:26.844196 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.844108 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 19:52:26.845850 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.845824 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-796b7c8599-d2vtr"] Nov 02 19:52:26.845981 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.845929 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 19:52:26.860311 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.858725 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/114df931-a2e1-4569-95c9-b7cc0c440079-metrics-server-audit-profiles\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.860311 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.858782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-secret-metrics-server-tls\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.860311 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.858923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-client-ca-bundle\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.860311 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.858969 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-secret-metrics-server-client-certs\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.860311 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.859033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/114df931-a2e1-4569-95c9-b7cc0c440079-audit-log\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.860311 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.859081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/114df931-a2e1-4569-95c9-b7cc0c440079-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.860311 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.859127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2f4h\" (UniqueName: \"kubernetes.io/projected/114df931-a2e1-4569-95c9-b7cc0c440079-kube-api-access-j2f4h\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.942181 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.942153 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dnc86\"" Nov 02 19:52:26.959518 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.959478 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/114df931-a2e1-4569-95c9-b7cc0c440079-metrics-server-audit-profiles\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.959608 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.959530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-secret-metrics-server-tls\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.959608 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.959559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-client-ca-bundle\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.959608 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.959593 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-secret-metrics-server-client-certs\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.959743 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.959615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/114df931-a2e1-4569-95c9-b7cc0c440079-audit-log\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.959743 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.959651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/114df931-a2e1-4569-95c9-b7cc0c440079-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.959743 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.959688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2f4h\" (UniqueName: \"kubernetes.io/projected/114df931-a2e1-4569-95c9-b7cc0c440079-kube-api-access-j2f4h\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.960136 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.960105 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/114df931-a2e1-4569-95c9-b7cc0c440079-audit-log\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.960473 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.960452 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/114df931-a2e1-4569-95c9-b7cc0c440079-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.960590 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.960569 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/114df931-a2e1-4569-95c9-b7cc0c440079-metrics-server-audit-profiles\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.962120 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.962101 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-secret-metrics-server-tls\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.962281 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.962257 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-client-ca-bundle\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:26.962324 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:26.962293 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-secret-metrics-server-client-certs\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:27.014534 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.014501 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2f4h\" (UniqueName: \"kubernetes.io/projected/114df931-a2e1-4569-95c9-b7cc0c440079-kube-api-access-j2f4h\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:27.103784 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.103706 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:27.171194 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.171070 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5"] Nov 02 19:52:27.179714 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:27.179686 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002e660f_77e7_4384_a7ca_514cc40a4cb6.slice/crio-40db643d8f04126b0b7301e1a1e4bf4b530d7c17b75425d0595aee8b64ddbcfb WatchSource:0}: Error finding container 40db643d8f04126b0b7301e1a1e4bf4b530d7c17b75425d0595aee8b64ddbcfb: Status 404 returned error can't find the container with id 40db643d8f04126b0b7301e1a1e4bf4b530d7c17b75425d0595aee8b64ddbcfb Nov 02 19:52:27.361973 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.361870 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:27.361973 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.361953 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:27.366757 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.366735 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:27.436625 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.436600 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:27.483020 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.482987 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-796b7c8599-d2vtr"] Nov 02 19:52:27.491951 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:27.491912 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114df931_a2e1_4569_95c9_b7cc0c440079.slice/crio-744fd9bce60a29357fe1ade982d425ed1b4677f9d68a08754b505689f0a865f2 WatchSource:0}: Error finding container 744fd9bce60a29357fe1ade982d425ed1b4677f9d68a08754b505689f0a865f2: Status 404 returned error can't find the container with id 744fd9bce60a29357fe1ade982d425ed1b4677f9d68a08754b505689f0a865f2 Nov 02 19:52:27.538895 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.538844 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" event={"ID":"114df931-a2e1-4569-95c9-b7cc0c440079","Type":"ContainerStarted","Data":"744fd9bce60a29357fe1ade982d425ed1b4677f9d68a08754b505689f0a865f2"} Nov 02 19:52:27.539992 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.539967 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" event={"ID":"002e660f-77e7-4384-a7ca-514cc40a4cb6","Type":"ContainerStarted","Data":"40db643d8f04126b0b7301e1a1e4bf4b530d7c17b75425d0595aee8b64ddbcfb"} Nov 02 19:52:27.545375 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.545353 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:27.934208 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.934172 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67c8cb984d-5gwb9"] Nov 02 19:52:27.938614 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.938591 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:27.969210 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.969184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-trusted-ca-bundle\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:27.969346 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.969227 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-service-ca\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:27.969415 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.969365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc678\" (UniqueName: \"kubernetes.io/projected/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-kube-api-access-qc678\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:27.969478 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.969445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-config\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:27.969478 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.969473 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-tmp\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:27.969573 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.969495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-serving-cert\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:27.969573 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.969523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-oauth-config\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:27.969681 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.969650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-oauth-serving-cert\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:27.990797 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:27.990773 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67c8cb984d-5gwb9"] Nov 02 19:52:28.038506 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.038472 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 19:52:28.070562 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.070529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-tmp\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.070562 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.070566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-serving-cert\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.070787 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.070592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-oauth-config\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.070787 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.070611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-oauth-serving-cert\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.070787 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.070651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-trusted-ca-bundle\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.070787 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.070679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-service-ca\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.070787 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.070721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc678\" (UniqueName: \"kubernetes.io/projected/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-kube-api-access-qc678\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.070787 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.070765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-config\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.071097 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.070981 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-tmp\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.071416 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.071387 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-oauth-serving-cert\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.071548 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.071525 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-service-ca\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.071589 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.071559 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-config\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.071621 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.071609 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-trusted-ca-bundle\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.073140 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.073114 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-oauth-config\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.073276 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.073258 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-serving-cert\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.123925 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.123870 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc678\" (UniqueName: \"kubernetes.io/projected/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-kube-api-access-qc678\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.249532 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.249454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:28.619668 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:28.619633 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67c8cb984d-5gwb9"] Nov 02 19:52:28.628502 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:28.628469 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ebae9e_27da_4d11_93c6_acc7657cd4d5.slice/crio-1aa740e4cd6c008dcabbca2e4cdf195848a227921fde726f1ba39f42d5381d92 WatchSource:0}: Error finding container 1aa740e4cd6c008dcabbca2e4cdf195848a227921fde726f1ba39f42d5381d92: Status 404 returned error can't find the container with id 1aa740e4cd6c008dcabbca2e4cdf195848a227921fde726f1ba39f42d5381d92 Nov 02 19:52:29.485388 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:29.485353 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:29.550483 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:29.550448 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c8cb984d-5gwb9" event={"ID":"f2ebae9e-27da-4d11-93c6-acc7657cd4d5","Type":"ContainerStarted","Data":"75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7"} Nov 02 19:52:29.550607 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:29.550510 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c8cb984d-5gwb9" event={"ID":"f2ebae9e-27da-4d11-93c6-acc7657cd4d5","Type":"ContainerStarted","Data":"1aa740e4cd6c008dcabbca2e4cdf195848a227921fde726f1ba39f42d5381d92"} Nov 02 19:52:29.555928 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:29.555746 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" event={"ID":"367d4910-a962-43b9-a7e6-a58b47636a24","Type":"ContainerStarted","Data":"8dc81addffc6752cc1c1b5166b21cb509facccc6a9fd946c6bfb505c30c5782c"} Nov 02 19:52:29.555928 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:29.555821 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" event={"ID":"367d4910-a962-43b9-a7e6-a58b47636a24","Type":"ContainerStarted","Data":"89861cb38ddde20d5b9d958012c87b913755daf918489d91f7a2f3c57be3d860"} Nov 02 19:52:29.555928 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:29.555858 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" event={"ID":"367d4910-a962-43b9-a7e6-a58b47636a24","Type":"ContainerStarted","Data":"3aa4453e3954660f67d4f3ce39d94c1df06ffcec86103948f0ecd03bad10fb0d"} Nov 02 19:52:29.658818 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:29.658758 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67c8cb984d-5gwb9" podStartSLOduration=2.658739943 podStartE2EDuration="2.658739943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:29.657845656 +0000 UTC m=+58.886813610" watchObservedRunningTime="2026-11-02 19:52:29.658739943 +0000 UTC m=+58.887707896" Nov 02 19:52:29.669613 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:29.669584 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:30.561579 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:30.561533 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" event={"ID":"114df931-a2e1-4569-95c9-b7cc0c440079","Type":"ContainerStarted","Data":"4bc25249ebc82898f2189d8e235ea7debb3dc09010eeebe5466ea3afb88cc462"} Nov 02 19:52:30.563762 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:30.563739 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" event={"ID":"002e660f-77e7-4384-a7ca-514cc40a4cb6","Type":"ContainerStarted","Data":"915c65eb8f0e42ac90fde6c094fbd8898802ee110b7fdb07cb370ac212aa49c4"} Nov 02 19:52:30.563997 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:30.563976 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:30.569012 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:30.568988 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:30.683597 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:30.683539 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" podStartSLOduration=2.6666648090000002 podStartE2EDuration="4.683520204s" totalImagesPullingTime="2.016855395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:30.682238642 +0000 UTC m=+59.911206597" watchObservedRunningTime="2026-11-02 19:52:30.683520204 +0000 UTC m=+59.912488159" Nov 02 19:52:30.787829 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:30.787734 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" podStartSLOduration=2.007852725 podStartE2EDuration="4.78772088s" totalImagesPullingTime="2.779868155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:30.786970065 +0000 UTC m=+60.015938020" watchObservedRunningTime="2026-11-02 19:52:30.78772088 +0000 UTC m=+60.016688833" Nov 02 19:52:30.873271 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:30.873236 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:31.509635 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:31.509605 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:31.569490 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:31.569457 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" event={"ID":"367d4910-a962-43b9-a7e6-a58b47636a24","Type":"ContainerStarted","Data":"26fdea211e0ec2619f037a3ee94ada7be90cac49911832f3e7aa9e3534c93b4d"} Nov 02 19:52:31.569644 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:31.569519 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" event={"ID":"367d4910-a962-43b9-a7e6-a58b47636a24","Type":"ContainerStarted","Data":"f7c1b1da826c528d8bb3d5d5bb987f914bdefafac0a1b01d3bdea40e523f8ecc"} Nov 02 19:52:31.569644 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:31.569545 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" event={"ID":"367d4910-a962-43b9-a7e6-a58b47636a24","Type":"ContainerStarted","Data":"e4c0e906888f25bea3019f994695065480fed6d7cf860c8c4b219fc319fa0883"} Nov 02 19:52:31.730302 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:31.730251 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" podStartSLOduration=2.674358688 podStartE2EDuration="7.730237003s" totalImagesPullingTime="5.055878315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:31.728686271 +0000 UTC m=+60.957654224" watchObservedRunningTime="2026-11-02 19:52:31.730237003 +0000 UTC m=+60.959204956" Nov 02 19:52:32.572962 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:32.572934 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:32.886755 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:32.886679 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:33.062221 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:33.062193 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:33.385343 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:33.385312 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:33.552305 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:33.552270 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:34.167449 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:34.167411 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:34.341560 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:34.341528 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:35.765638 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:35.765604 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:35.953196 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:35.953160 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:36.120129 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:36.120054 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:37.053763 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.053727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:37.054280 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.053780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:37.147923 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.147892 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:52:37.148040 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.147902 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:52:37.156458 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.156429 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:37.156458 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.156443 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:37.218225 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.218197 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sk47\"" Nov 02 19:52:37.221376 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.221360 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:37.223641 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.223628 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gf4xf\"" Nov 02 19:52:37.226315 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.226300 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:37.254611 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.254573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwzr\" (UniqueName: \"kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr\") pod \"network-check-target-m2tvk\" (UID: \"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9\") " pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:37.303069 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.303046 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:52:37.352787 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.352718 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:52:37.358097 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.358078 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwzr\" (UniqueName: \"kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr\") pod \"network-check-target-m2tvk\" (UID: \"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9\") " pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:37.528204 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.528177 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ltmn5\"" Nov 02 19:52:37.531242 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.531226 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:37.593330 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.593304 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r67nn"] Nov 02 19:52:37.602578 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:37.602550 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d60e019_846f_497e_a1fb_1c972295523b.slice/crio-459f419345c9e0095001a97e900a384a3394c085d5754d58406123ddc91eef11 WatchSource:0}: Error finding container 459f419345c9e0095001a97e900a384a3394c085d5754d58406123ddc91eef11: Status 404 returned error can't find the container with id 459f419345c9e0095001a97e900a384a3394c085d5754d58406123ddc91eef11 Nov 02 19:52:37.645370 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.645341 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4bpmh"] Nov 02 19:52:37.653248 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:37.653225 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dde0e70_73a7_4198_831e_875401c4c3cd.slice/crio-57e961e28fea58c3461f0ce6e52c6bac9a40fdfa7581f49eacea6a3fb7adabcd WatchSource:0}: Error finding container 57e961e28fea58c3461f0ce6e52c6bac9a40fdfa7581f49eacea6a3fb7adabcd: Status 404 returned error can't find the container with id 57e961e28fea58c3461f0ce6e52c6bac9a40fdfa7581f49eacea6a3fb7adabcd Nov 02 19:52:37.897613 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:37.897533 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2tvk"] Nov 02 19:52:37.905513 ip-10-0-134-194 kubenswrapper[2575]: W1102 19:52:37.905482 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ff0bc2_a4cf_45f3_b870_2526d024b6c9.slice/crio-02867403e1c69adf635a3633653707f6adc03ce1e7eb96c99d50185c71b76e3d WatchSource:0}: Error finding container 02867403e1c69adf635a3633653707f6adc03ce1e7eb96c99d50185c71b76e3d: Status 404 returned error can't find the container with id 02867403e1c69adf635a3633653707f6adc03ce1e7eb96c99d50185c71b76e3d Nov 02 19:52:38.250211 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:38.250176 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:38.250637 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:38.250236 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:38.254400 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:38.254382 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:38.581642 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:38.581580 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:38.593057 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:38.593028 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4bpmh" event={"ID":"6dde0e70-73a7-4198-831e-875401c4c3cd","Type":"ContainerStarted","Data":"57e961e28fea58c3461f0ce6e52c6bac9a40fdfa7581f49eacea6a3fb7adabcd"} Nov 02 19:52:38.594294 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:38.594270 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r67nn" event={"ID":"0d60e019-846f-497e-a1fb-1c972295523b","Type":"ContainerStarted","Data":"459f419345c9e0095001a97e900a384a3394c085d5754d58406123ddc91eef11"} Nov 02 19:52:38.595273 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:38.595251 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2tvk" event={"ID":"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9","Type":"ContainerStarted","Data":"02867403e1c69adf635a3633653707f6adc03ce1e7eb96c99d50185c71b76e3d"} Nov 02 19:52:38.599217 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:38.599201 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:38.626840 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:38.626815 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:38.809141 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:38.809108 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:38.831278 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:38.831248 2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8478dfbff8-ldrtw"] Nov 02 19:52:39.602183 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:39.602092 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r67nn" event={"ID":"0d60e019-846f-497e-a1fb-1c972295523b","Type":"ContainerStarted","Data":"da67f420c371a02767179108fbb15f26acfaeac105e1e787ea8643cb85a4404d"} Nov 02 19:52:39.602183 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:39.602158 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r67nn" event={"ID":"0d60e019-846f-497e-a1fb-1c972295523b","Type":"ContainerStarted","Data":"33ed92a19851ff4efbf878ea88ab18e4934641f4e2ae80cc4d0848e9fde85050"} Nov 02 19:52:40.214325 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:40.214284 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:40.389069 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:40.389035 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:40.764834 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:40.764795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:40.963681 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:40.963645 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:41.747723 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:41.747688 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:41.926112 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:41.926080 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:42.615246 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:42.615207 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2tvk" event={"ID":"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9","Type":"ContainerStarted","Data":"03737a576153fc8c10255f949f92c2aa8a7ad5496a6f3957583f6717b351bd16"} Nov 02 19:52:42.615428 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:42.615288 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:42.721991 ip-10-0-134-194 kubenswrapper[2575]: I1102 19:52:42.721934 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r67nn" podStartSLOduration=69.056062578 podStartE2EDuration="1m10.721914553s" totalImagesPullingTime="1.665851975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:39.709158921 +0000 UTC m=+68.938126876" watchObservedRunningTime="2026-11-02 19:52:42.721914553 +0000 UTC m=+71.950882508" Nov 02 19:52:43.120106 ip-10-0-134-194 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 19:52:43.137229 ip-10-0-134-194 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 19:52:43.137444 ip-10-0-134-194 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 19:52:43.137645 ip-10-0-134-194 systemd[1]: kubelet.service: Consumed 4.215s CPU time. Nov 02 19:52:43.145544 ip-10-0-134-194 systemd[1]: Starting Kubernetes Kubelet... Nov 02 19:52:43.225656 ip-10-0-134-194 kubenswrapper[5561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:52:43.225656 ip-10-0-134-194 kubenswrapper[5561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 19:52:43.225656 ip-10-0-134-194 kubenswrapper[5561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:52:43.226276 ip-10-0-134-194 kubenswrapper[5561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:52:43.227702 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227619 5561 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:43.227702 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227702 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227706 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227709 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227712 5561 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227715 5561 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227718 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227721 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227723 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227726 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227730 5561 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227732 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227735 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227738 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227740 5561 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227747 5561 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227750 5561 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227753 5561 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227755 5561 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227758 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227761 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:43.227773 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227763 5561 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227766 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227770 5561 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227775 5561 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227779 5561 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227782 5561 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227786 5561 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227790 5561 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227793 5561 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227796 5561 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227799 5561 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227802 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227805 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227807 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227810 5561 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227812 5561 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227815 5561 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227817 5561 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227820 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:43.228313 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227823 5561 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227826 5561 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227828 5561 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227831 5561 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227833 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227836 5561 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227839 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227841 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227845 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227847 5561 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227853 5561 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227856 5561 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227860 5561 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227862 5561 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227865 5561 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227868 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227890 5561 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227896 5561 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227901 5561 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227905 5561 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:43.228780 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227909 5561 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227912 5561 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227915 5561 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227918 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227921 5561 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227924 5561 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227926 5561 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227929 5561 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227932 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227934 5561 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227937 5561 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227939 5561 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227942 5561 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227945 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227947 5561 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227950 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227953 5561 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227955 5561 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227958 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:43.229305 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227960 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227963 5561 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227966 5561 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227972 5561 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227975 5561 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227978 5561 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227980 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227983 5561 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227986 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227989 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227992 5561 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227994 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.227997 5561 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228000 5561 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228002 5561 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228005 5561 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228007 5561 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228010 5561 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228012 5561 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228015 5561 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:43.229812 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228018 5561 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228020 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228024 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228027 5561 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228029 5561 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228032 5561 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228034 5561 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228897 5561 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228905 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228909 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228913 5561 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228917 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228920 5561 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228923 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228925 5561 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228935 5561 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228939 5561 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228941 5561 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228944 5561 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:43.230353 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228947 5561 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228950 5561 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228952 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228955 5561 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228957 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228960 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228962 5561 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228965 5561 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228968 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228970 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228972 5561 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228975 5561 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228977 5561 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228980 5561 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228982 5561 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228985 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228987 5561 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228990 5561 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228992 5561 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:43.230890 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228995 5561 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.228999 5561 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229003 5561 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229005 5561 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229008 5561 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229011 5561 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229014 5561 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229016 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229019 5561 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229022 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229030 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229033 5561 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229036 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229038 5561 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229041 5561 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229044 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229053 5561 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229055 5561 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229058 5561 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:43.231510 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229060 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229063 5561 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229065 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229068 5561 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229071 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229074 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229076 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229079 5561 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229082 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229084 5561 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229087 5561 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229089 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229092 5561 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229095 5561 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229097 5561 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229099 5561 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229102 5561 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229104 5561 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229107 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229109 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:43.232017 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229112 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229114 5561 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229117 5561 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229120 5561 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229130 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229133 5561 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229136 5561 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229138 5561 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229141 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229143 5561 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229146 5561 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229149 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229151 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229154 5561 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229156 5561 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229159 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229162 5561 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229165 5561 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229167 5561 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229170 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:43.232512 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229172 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229175 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229177 5561 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229180 5561 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229182 5561 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229187 5561 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229191 5561 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229194 5561 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229197 5561 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229200 5561 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229202 5561 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229205 5561 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229208 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229210 5561 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229213 5561 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229215 5561 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229778 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229784 5561 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229789 5561 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:43.233018 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229794 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229797 5561 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229800 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229805 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229807 5561 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229810 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229813 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229816 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229819 5561 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229822 5561 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229825 5561 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229828 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229830 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229833 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229836 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229839 5561 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229841 5561 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229844 5561 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229846 5561 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229849 5561 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:43.233487 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229852 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229854 5561 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229857 5561 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229859 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229862 5561 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229864 5561 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229867 5561 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229869 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229885 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229888 5561 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229891 5561 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229894 5561 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229896 5561 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229900 5561 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229903 5561 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229908 5561 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229910 5561 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229913 5561 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229915 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:43.234020 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229918 5561 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229921 5561 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229923 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229926 5561 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229928 5561 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229931 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229933 5561 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229937 5561 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229939 5561 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229942 5561 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229945 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229948 5561 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229951 5561 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229953 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229956 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229959 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229961 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229964 5561 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229966 5561 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229969 5561 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:43.234494 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229971 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229974 5561 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229976 5561 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229979 5561 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229982 5561 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229984 5561 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229987 5561 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229989 5561 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229993 5561 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229996 5561 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.229999 5561 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230001 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230004 5561 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230006 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230009 5561 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230011 5561 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230014 5561 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230016 5561 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230019 5561 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230022 5561 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:43.235016 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230024 5561 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230027 5561 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230030 5561 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230033 5561 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230035 5561 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230038 5561 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230040 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230043 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230045 5561 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230048 5561 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230050 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230053 5561 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230055 5561 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230058 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230061 5561 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230063 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230066 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230068 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230071 5561 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:43.235794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230074 5561 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230078 5561 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230081 5561 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230084 5561 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230087 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230180 5561 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230187 5561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230193 5561 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230197 5561 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230202 5561 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230205 5561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230210 5561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230214 5561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230217 5561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230220 5561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230224 5561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230229 5561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230232 5561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230235 5561 flags.go:64] FLAG: --cgroup-root="" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230238 5561 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230241 5561 flags.go:64] FLAG: --client-ca-file="" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230244 5561 flags.go:64] FLAG: --cloud-provider="external" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230247 5561 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 19:52:43.236485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230252 5561 flags.go:64] FLAG: --cluster-domain="" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230254 5561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230258 5561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230261 5561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230264 5561 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230268 5561 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230271 5561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230274 5561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230278 5561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230281 5561 flags.go:64] FLAG: --contention-profiling="false" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230285 5561 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230288 5561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230292 5561 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230295 5561 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230299 5561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230303 5561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230306 5561 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230308 5561 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230311 5561 flags.go:64] FLAG: --enable-server="true" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230314 5561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230318 5561 flags.go:64] FLAG: --event-burst="100" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230321 5561 flags.go:64] FLAG: --event-qps="50" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230325 5561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230328 5561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 19:52:43.237351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230331 5561 flags.go:64] FLAG: --eviction-hard="" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230337 5561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230340 5561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230343 5561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230346 5561 flags.go:64] FLAG: --eviction-soft="" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230349 5561 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230353 5561 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230356 5561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230359 5561 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230362 5561 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230365 5561 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230368 5561 flags.go:64] FLAG: --feature-gates="" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230372 5561 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230376 5561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230379 5561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230382 5561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230385 5561 flags.go:64] FLAG: --healthz-port="10248" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230391 5561 flags.go:64] FLAG: --help="false" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230396 5561 flags.go:64] FLAG: --hostname-override="ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230399 5561 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230402 5561 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230405 5561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230408 5561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 19:52:43.237979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230412 5561 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230415 5561 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230418 5561 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230420 5561 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230423 5561 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230426 5561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230430 5561 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230433 5561 flags.go:64] FLAG: --kube-reserved="" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230436 5561 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230438 5561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230442 5561 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230445 5561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230449 5561 flags.go:64] FLAG: --lock-file="" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230452 5561 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230455 5561 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230458 5561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230463 5561 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230466 5561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230469 5561 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230472 5561 flags.go:64] FLAG: --logging-format="text" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230475 5561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230478 5561 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230481 5561 flags.go:64] FLAG: --manifest-url="" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230484 5561 flags.go:64] FLAG: --manifest-url-header="" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230488 5561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 19:52:43.238543 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230491 5561 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230495 5561 flags.go:64] FLAG: --max-pods="110" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230500 5561 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230504 5561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230507 5561 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230509 5561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230512 5561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230515 5561 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230518 5561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230525 5561 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230528 5561 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230532 5561 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230535 5561 flags.go:64] FLAG: --pod-cidr="" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230538 5561 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230541 5561 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230544 5561 flags.go:64] FLAG: --pods-per-core="0" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230547 5561 flags.go:64] FLAG: --port="10250" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230551 5561 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230554 5561 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-012c7be788c730f37" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230557 5561 flags.go:64] FLAG: --qos-reserved="" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230560 5561 flags.go:64] FLAG: --read-only-port="10255" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230563 5561 flags.go:64] FLAG: --register-node="true" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230566 5561 flags.go:64] FLAG: --register-with-taints="" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230570 5561 flags.go:64] FLAG: --registry-burst="10" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230573 5561 flags.go:64] FLAG: --registry-qps="5" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230576 5561 flags.go:64] FLAG: --reserved-cpus="" Nov 02 19:52:43.239221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230579 5561 flags.go:64] FLAG: --reserved-memory="" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230583 5561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230586 5561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230589 5561 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230592 5561 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230594 5561 flags.go:64] FLAG: --runonce="false" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230597 5561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230600 5561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230605 5561 flags.go:64] FLAG: --seccomp-default="false" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230609 5561 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230612 5561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230616 5561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230619 5561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230622 5561 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230625 5561 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230629 5561 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230631 5561 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230634 5561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230637 5561 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230640 5561 flags.go:64] FLAG: --system-cgroups="" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230643 5561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230648 5561 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230651 5561 flags.go:64] FLAG: --tls-cert-file="" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230654 5561 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230659 5561 flags.go:64] FLAG: --tls-min-version="" Nov 02 19:52:43.239896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230662 5561 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230665 5561 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230667 5561 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230670 5561 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230673 5561 flags.go:64] FLAG: --v="2" Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230677 5561 flags.go:64] FLAG: --version="false" Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230681 5561 flags.go:64] FLAG: --vmodule="" Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230685 5561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.230689 5561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230821 5561 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230824 5561 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230827 5561 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230830 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230833 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230835 5561 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230840 5561 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230842 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230846 5561 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230849 5561 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230851 5561 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230854 5561 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230856 5561 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:43.240584 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230859 5561 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230861 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230864 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230867 5561 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230870 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230887 5561 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230889 5561 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230892 5561 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230895 5561 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230898 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230900 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230905 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230910 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230914 5561 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230919 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230923 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230927 5561 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230931 5561 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230935 5561 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:43.241137 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230939 5561 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230944 5561 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230947 5561 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230950 5561 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230952 5561 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230955 5561 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230960 5561 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230963 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230969 5561 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230974 5561 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230978 5561 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230981 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230984 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230986 5561 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230989 5561 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230992 5561 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230994 5561 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.230997 5561 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231000 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:43.241692 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231002 5561 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231005 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231008 5561 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231010 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231013 5561 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231015 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231018 5561 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231021 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231024 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231027 5561 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231031 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231035 5561 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231039 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231044 5561 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231048 5561 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231052 5561 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231058 5561 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231061 5561 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231064 5561 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231068 5561 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:43.242347 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231070 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231075 5561 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231078 5561 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231081 5561 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231084 5561 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231087 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231090 5561 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231092 5561 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231095 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231098 5561 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231100 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231103 5561 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231105 5561 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231108 5561 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231111 5561 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231115 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231119 5561 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231123 5561 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231127 5561 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231132 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:43.243419 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231135 5561 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231139 5561 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231143 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231146 5561 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231149 5561 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231152 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231154 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231157 5561 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231159 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231162 5561 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231164 5561 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231168 5561 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231171 5561 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231175 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231178 5561 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:43.244604 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.231206 5561 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231251 5561 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231255 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231258 5561 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231261 5561 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231264 5561 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231266 5561 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231269 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231271 5561 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231274 5561 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231277 5561 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231279 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231282 5561 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231286 5561 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231289 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231292 5561 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231294 5561 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231297 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231300 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:43.245127 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231302 5561 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231305 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231307 5561 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231310 5561 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231312 5561 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231316 5561 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231321 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231326 5561 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231329 5561 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231332 5561 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231337 5561 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231341 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231344 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231347 5561 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231349 5561 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231352 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231355 5561 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231358 5561 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231361 5561 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231363 5561 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:43.245769 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231366 5561 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231368 5561 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231371 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231374 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231376 5561 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231379 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231381 5561 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231384 5561 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231386 5561 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231390 5561 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231394 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231398 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231402 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231406 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231408 5561 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231411 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231413 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231416 5561 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231419 5561 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:43.246346 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231421 5561 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231424 5561 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231427 5561 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231431 5561 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231435 5561 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231438 5561 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231441 5561 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231443 5561 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231446 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231451 5561 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231454 5561 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231457 5561 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231460 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231463 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231465 5561 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231468 5561 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231471 5561 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231473 5561 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231476 5561 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231478 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:43.246807 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231481 5561 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231483 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231486 5561 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231488 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231491 5561 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231493 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231496 5561 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231499 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231501 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231504 5561 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231507 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231510 5561 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231513 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231515 5561 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231518 5561 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231522 5561 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231526 5561 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231529 5561 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231531 5561 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231534 5561 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:43.247316 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231536 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231539 5561 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231542 5561 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231545 5561 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231548 5561 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231550 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231553 5561 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231555 5561 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.231558 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.237376 5561 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.237394 5561 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.237419 5561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.237424 5561 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237483 5561 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237488 5561 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237491 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237494 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237497 5561 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237500 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:43.247796 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237503 5561 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237505 5561 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237508 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237511 5561 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237514 5561 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237517 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237520 5561 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237523 5561 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237526 5561 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237529 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237532 5561 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237535 5561 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237538 5561 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237540 5561 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237543 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237545 5561 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237548 5561 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237551 5561 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237554 5561 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:43.248282 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237556 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237560 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237563 5561 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237565 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237568 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237571 5561 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237573 5561 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237576 5561 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237580 5561 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237582 5561 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237585 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237588 5561 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237591 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237593 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237596 5561 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237598 5561 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237601 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237604 5561 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237607 5561 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237609 5561 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:43.248794 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237611 5561 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237620 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237624 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237627 5561 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237629 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237632 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237634 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237637 5561 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237640 5561 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237643 5561 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237647 5561 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237650 5561 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237654 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237656 5561 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237659 5561 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237662 5561 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237665 5561 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237667 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237670 5561 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237673 5561 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:43.249441 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237675 5561 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237678 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237680 5561 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237683 5561 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237685 5561 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237688 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237690 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237692 5561 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237695 5561 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237698 5561 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237701 5561 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237703 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237706 5561 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237709 5561 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237718 5561 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237729 5561 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237732 5561 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237734 5561 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237737 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237740 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:43.250114 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237742 5561 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237745 5561 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237748 5561 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237752 5561 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237755 5561 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237758 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237761 5561 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237764 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237767 5561 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237769 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237772 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237775 5561 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237778 5561 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237780 5561 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237783 5561 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237785 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237788 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237791 5561 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237794 5561 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:43.250826 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237796 5561 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237799 5561 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.237829 5561 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237988 5561 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237994 5561 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.237998 5561 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238001 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238004 5561 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238014 5561 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238017 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238020 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238022 5561 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238025 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238028 5561 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238031 5561 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238033 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:43.251425 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238036 5561 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238038 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238041 5561 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238044 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238046 5561 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238049 5561 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238051 5561 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238054 5561 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238056 5561 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238059 5561 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238062 5561 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238064 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238067 5561 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238069 5561 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238072 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238074 5561 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238077 5561 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238080 5561 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238082 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238085 5561 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:43.252120 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238087 5561 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238090 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238092 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238095 5561 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238098 5561 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238102 5561 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238104 5561 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238107 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238109 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238112 5561 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238114 5561 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238117 5561 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238119 5561 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238122 5561 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238124 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238127 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238129 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238132 5561 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238134 5561 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238137 5561 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:43.252857 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238139 5561 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238142 5561 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238144 5561 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238147 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238149 5561 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238152 5561 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238155 5561 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238158 5561 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238160 5561 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238163 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238165 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238168 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238171 5561 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238173 5561 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238176 5561 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238178 5561 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238181 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238184 5561 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238188 5561 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:43.253431 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238190 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238192 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238195 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238198 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238200 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238203 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238205 5561 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238208 5561 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238210 5561 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238213 5561 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238216 5561 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238237 5561 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238241 5561 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238245 5561 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238248 5561 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238251 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238254 5561 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238257 5561 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238262 5561 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:43.253955 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238265 5561 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238268 5561 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238271 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238273 5561 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238276 5561 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238279 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238282 5561 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238284 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238287 5561 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238290 5561 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238292 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238295 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238298 5561 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238303 5561 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238307 5561 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.238332 5561 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:52:43.254432 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238364 5561 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238368 5561 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238371 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238374 5561 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238377 5561 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238380 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238382 5561 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238385 5561 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238387 5561 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238390 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238393 5561 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238395 5561 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238398 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238400 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238403 5561 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238407 5561 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238410 5561 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238412 5561 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238415 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238417 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:43.254829 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238420 5561 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238423 5561 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238425 5561 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238428 5561 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238430 5561 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238433 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238435 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238438 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238440 5561 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238443 5561 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238446 5561 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238448 5561 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238451 5561 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238454 5561 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238456 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238459 5561 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238461 5561 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238463 5561 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238466 5561 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:43.255382 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238468 5561 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238471 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238474 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238478 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238480 5561 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238483 5561 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238486 5561 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238488 5561 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238493 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238495 5561 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238498 5561 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238500 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238503 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238506 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238508 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238511 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238513 5561 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238516 5561 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238519 5561 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:43.255840 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238521 5561 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238524 5561 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238528 5561 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238532 5561 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238535 5561 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238540 5561 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238543 5561 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238546 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238549 5561 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238552 5561 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238554 5561 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238557 5561 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238560 5561 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238562 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238565 5561 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238568 5561 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238573 5561 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238576 5561 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238578 5561 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:43.256312 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238581 5561 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238584 5561 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238588 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238590 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238594 5561 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238598 5561 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238601 5561 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238604 5561 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238607 5561 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238610 5561 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238613 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238616 5561 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238619 5561 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238622 5561 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238625 5561 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238627 5561 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238630 5561 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238634 5561 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238636 5561 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:43.256820 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238640 5561 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238643 5561 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238645 5561 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238648 5561 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238651 5561 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238653 5561 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238656 5561 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238658 5561 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238661 5561 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:52:43.238663 5561 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.238722 5561 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.239698 5561 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.239749 5561 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.239932 5561 server.go:1014] "Starting client certificate rotation" Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.240093 5561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.240144 5561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:46:32 +0000 UTC" deadline="2028-06-23 17:39:00.879464158 +0000 UTC" Nov 02 19:52:43.257387 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.240181 5561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14373h46m17.639286974s" Nov 02 19:52:43.257787 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.240437 5561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:52:43.257787 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.241536 5561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:52:43.257787 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.242844 5561 log.go:25] "Validated CRI v1 runtime API" Nov 02 19:52:43.257787 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.244819 5561 log.go:25] "Validated CRI v1 image API" Nov 02 19:52:43.257787 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.245523 5561 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 19:52:43.257787 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.247711 5561 fs.go:135] Filesystem UUIDs: map[0667c6c9-0462-4dfd-afc2-998be80ed013:/dev/nvme0n1p4 6135173a-d3d7-4366-a317-e20d1c9719ea:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 02 19:52:43.257988 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.247732 5561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02867403e1c69adf635a3633653707f6adc03ce1e7eb96c99d50185c71b76e3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02867403e1c69adf635a3633653707f6adc03ce1e7eb96c99d50185c71b76e3d/userdata/shm major:0 minor:414 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02fc097a077a44cf61b61260e0a9c49039a5fb31363458ad74e8dd86c890b543/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02fc097a077a44cf61b61260e0a9c49039a5fb31363458ad74e8dd86c890b543/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a2a3a7bb67cda384d06d04ecde62c7e5e7c574c2712ebc9f2798a5fae4aac8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a2a3a7bb67cda384d06d04ecde62c7e5e7c574c2712ebc9f2798a5fae4aac8e/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1aa740e4cd6c008dcabbca2e4cdf195848a227921fde726f1ba39f42d5381d92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1aa740e4cd6c008dcabbca2e4cdf195848a227921fde726f1ba39f42d5381d92/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dde184d4c22da9ef2897be99a6e57bd6448f62f35cad908c9e83dadb882b4eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dde184d4c22da9ef2897be99a6e57bd6448f62f35cad908c9e83dadb882b4eb/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/263cb0b0be40e4059ac1a4f5018e966d5ee3bfd6a417af4702b2846d7ed335d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/263cb0b0be40e4059ac1a4f5018e966d5ee3bfd6a417af4702b2846d7ed335d8/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f28cfd977351c65133951958224c2e26bfcd8673d78cb02f2e20d41e0c6b978/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f28cfd977351c65133951958224c2e26bfcd8673d78cb02f2e20d41e0c6b978/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31198e0b2a84a17c7d0d20a0ff5f15365a71d4ef25070fea99b96b564632a206/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31198e0b2a84a17c7d0d20a0ff5f15365a71d4ef25070fea99b96b564632a206/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d39bbb7bf19d99d8016b542db8d07a6d84fb326f5c9f5610a1768ccca21a035/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d39bbb7bf19d99d8016b542db8d07a6d84fb326f5c9f5610a1768ccca21a035/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40db643d8f04126b0b7301e1a1e4bf4b530d7c17b75425d0595aee8b64ddbcfb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40db643d8f04126b0b7301e1a1e4bf4b530d7c17b75425d0595aee8b64ddbcfb/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/459f419345c9e0095001a97e900a384a3394c085d5754d58406123ddc91eef11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/459f419345c9e0095001a97e900a384a3394c085d5754d58406123ddc91eef11/userdata/shm major:0 minor:409 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57e961e28fea58c3461f0ce6e52c6bac9a40fdfa7581f49eacea6a3fb7adabcd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57e961e28fea58c3461f0ce6e52c6bac9a40fdfa7581f49eacea6a3fb7adabcd/userdata/shm major:0 minor:411 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58558a20b85541cdf8b2dbde0ee288393a098121a06a7d15ad31d4ad86139082/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58558a20b85541cdf8b2dbde0ee288393a098121a06a7d15ad31d4ad86139082/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f7105b2ce2bca00f516d0fcc551248c1ffada61b655810271ee923b37ea20ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f7105b2ce2bca00f516d0fcc551248c1ffada61b655810271ee923b37ea20ad/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ecaa60df66aea858c5e29edde6204dd9e9ac319b453223e0e3d2100da7b0294/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ecaa60df66aea858c5e29edde6204dd9e9ac319b453223e0e3d2100da7b0294/userdata/shm major:0 minor:276 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73f662b539aae6ba3dc20130eaddfea9caf3ca04e7fd2c4d52b6ad2477be4367/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73f662b539aae6ba3dc20130eaddfea9caf3ca04e7fd2c4d52b6ad2477be4367/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/744fd9bce60a29357fe1ade982d425ed1b4677f9d68a08754b505689f0a865f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/744fd9bce60a29357fe1ade982d425ed1b4677f9d68a08754b505689f0a865f2/userdata/shm major:0 minor:345 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9957bdb38b1f268e3b8c2ca16ad13609c65ae293836d9f9923f4c5a1e1e0826e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9957bdb38b1f268e3b8c2ca16ad13609c65ae293836d9f9923f4c5a1e1e0826e/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9972847e89925e9621b618b0a2bfd6449cd0e69e9cbc206944b2bcbc0cec80c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9972847e89925e9621b618b0a2bfd6449cd0e69e9cbc206944b2bcbc0cec80c3/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99f0054bcd118b2b376e5e18f7804638eb0cf21297f498e7c21d07305461efb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99f0054bcd118b2b376e5e18f7804638eb0cf21297f498e7c21d07305461efb7/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab5d393663b3d2aefb1aeaa801d18001114cadd1d6df0260dd98f18de4fdc46c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab5d393663b3d2aefb1aeaa801d18001114cadd1d6df0260dd98f18de4fdc46c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad4140c0ae0093f04b2e4d9ec3d2214f289e7f173039b984103ae84a8a89c560/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad4140c0ae0093f04b2e4d9ec3d2214f289e7f173039b984103ae84a8a89c560/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8cd86680e0044abba406a9e8aaa8f180955fdbcb150e85487e7246c730f853f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8cd86680e0044abba406a9e8aaa8f180955fdbcb150e85487e7246c730f853f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca8189f849c255c459632645fbb1e2159d535964b6fc61524994834c4b1f35df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca8189f849c255c459632645fbb1e2159d535964b6fc61524994834c4b1f35df/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/002e660f-77e7-4384-a7ca-514cc40a4cb6/volumes/kubernetes.io~projected/kube-api-access-mw92p:{mountpoint:/var/lib/kubelet/pods/002e660f-77e7-4384-a7ca-514cc40a4cb6/volumes/kubernetes.io~projected/kube-api-access-mw92p major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/002e660f-77e7-4384-a7ca-514cc40a4cb6/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/002e660f-77e7-4384-a7ca-514cc40a4cb6/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0170aa95-6bc5-4107-98f6-7beea4227f4f/volumes/kubernetes.io~projected/kube-api-access-9t966:{mountpoint:/var/lib/kubelet/pods/0170aa95-6bc5-4107-98f6-7beea4227f4f/volumes/kubernetes.io~projected/kube-api-access-9t966 major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0170aa95-6bc5-4107-98f6-7beea4227f4f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0170aa95-6bc5-4107-98f6-7beea4227f4f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0170aa95-6bc5-4107-98f6-7beea4227f4f/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0170aa95-6bc5-4107-98f6-7beea4227f4f/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/092fd489-4d42-46d1-a178-e95204a400e8/volumes/kubernetes.io~projected/kube-api-access-jmg6s:{mountpoint:/var/lib/kubelet/pods/092fd489-4d42-46d1-a178-e95204a400e8/volumes/kubernetes.io~projected/kube-api-access-jmg6s major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/092fd489-4d42-46d1-a178-e95204a400e8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/092fd489-4d42-46d1-a178-e95204a400e8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/092fd489-4d42-46d1-a178-e95204a400e8/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/092fd489-4d42-46d1-a178-e95204a400e8/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d60e019-846f-497e-a1fb-1c972295523b/volumes/kubernetes.io~projected/kube-api-access-s4j9w:{mountpoint:/var/lib/kubelet/pods/0d60e019-846f-497e-a1fb-1c972295523b/volumes/kubernetes.io~projected/kube-api-access-s4j9w major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d60e019-846f-497e-a1fb-1c972295523b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/0d60e019-846f-497e-a1fb-1c972295523b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/114df931-a2e1-4569-95c9-b7cc0c440079/volumes/kubernetes.io~projected/kube-api-access-j2f4h:{mountpoint:/var/lib/kubelet/pods/114df931-a2e1-4569-95c9-b7cc0c440079/volumes/kubernetes.io~projected/kube-api-access-j2f4h major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/114df931-a2e1-4569-95c9-b7cc0c440079/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/114df931-a2e1-4569-95c9-b7cc0c440079/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/114df931-a2e1-4569-95c9-b7cc0c440079/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/114df931-a2e1-4569-95c9-b7cc0c440079/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/114df931-a2e1-4569-95c9-b7cc0c440079/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/114df931-a2e1-4569-95c9-b7cc0c440079/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab821fc-d8d9-4ed9-86df-56979b6ac88a/volumes/kubernetes.io~projected/kube-api-access-wwzsj:{mountpoint:/var/lib/kubelet/pods/2ab821fc-d8d9-4ed9-86df-56979b6ac88a/volumes/kubernetes.io~projected/kube-api-access-wwzsj major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab821fc-d8d9-4ed9-86df-56979b6ac88a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2ab821fc-d8d9-4ed9-86df-56979b6ac88a/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e/volumes/kubernetes.io~projected/kube-api-access-q46jj:{mountpoint:/var/lib/kubelet/pods/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e/volumes/kubernetes.io~projected/kube-api-access-q46jj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~projected/kube-api-access-vknpr:{mountpoint:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~projected/kube-api-access-vknpr major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ff0392-2388-465d-94b5-c2839f6eac84/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/54ff0392-2388-465d-94b5-c2839f6eac84/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6dde0e70-73a7-4198-831e-875401c4c3cd/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6dde0e70-73a7-4198-831e-875401c4c3cd/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:408 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57/volumes/kubernetes.io~projected/kube-api-access-c87tb:{mountpoint:/var/lib/kubelet/pods/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57/volumes/kubernetes.io~projected/kube-api-access-c87tb major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98798482-f8b5-401e-9aa2-363cda917fba/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/98798482-f8b5-401e-9aa2-363cda917fba/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98798482-f8b5-401e-9aa2-363cda917fba/volumes/kubernetes.io~projected/kube-api-access-bb5kf:{mountpoint:/var/lib/kubelet/pods/98798482-f8b5-401e-9aa2-363cda917fba/volumes/kubernetes.io~projected/kube-api-access-bb5kf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98798482-f8b5-401e-9aa2-363cda917fba/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/98798482-f8b5-401e-9aa2-363cda917fba/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3823ef4-9aae-417c-bc9a-a04253589bf5/volumes/kubernetes.io~projected/kube-api-access-fgkrt:{mountpoint:/var/lib/kubelet/pods/a3823ef4-9aae-417c-bc9a-a04253589bf5/volumes/kubernetes.io~projected/kube-api-access-fgkrt major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a409cedc-aedc-4130-aad6-4405f1539799/volumes/kubernetes.io~projected/kube-api-access-9x8ws:{mountpoint:/var/lib/kubelet/pods/a409cedc-aedc-4130-aad6-4405f1539799/volumes/kubernetes.io~projected/kube-api-access-9x8ws major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a409cedc-aedc-4130-aad6-4405f1539799/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/a409cedc-aedc-4130-aad6-4405f1539799/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9/volumes/kubernetes.io~projected/kube-api-access-gnwzr:{mountpoint:/var/lib/kubelet/pods/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9/volumes/kubernetes.io~projected/kube-api-access-gnwzr major:0 minor:413 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c10c2efe-3e34-4ca7-b972-680cde439609/volumes/kubernetes.io~projected/kube-api-access-4v7m7:{mountpoint:/var/lib/kubelet/pods/c10c2efe-3e34-4ca7-b972-680cde439609/volumes/kubernetes.io~projected/kube-api-access-4v7m7 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5d86ef2-bb3e-4128-a81e-875fd9f6c527/volumes/kubernetes.io~projected/kube-api-access-xbxrc:{mountpoint:/var/lib/kubelet/pods/c5d86ef2-bb3e-4128-a81e-875fd9f6c527/volumes/kubernetes.io~projected/kube-api-access-xbxrc major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5d86ef2-bb3e-4128-a81e-875fd9f6c527/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c5d86ef2-bb3e-4128-a81e-875fd9f6c527/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5d86ef2-bb3e-4128-a81e-875fd9f6c527/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/c5d86ef2-bb3e-4128-a81e-875fd9f6c527/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10bb7fa-2eb9-4215-b467-c2165c4c76f5/volumes/kubernetes.io~projected/kube-api-access-s4j5s:{mountpoint:/var/lib/kubelet/pods/d10bb7fa-2eb9-4215-b467-c2165c4c76f5/volumes/kubernetes.io~projected/kube-api-access-s4j5s major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e588ae83-565a-4711-a7b7-d77b8ae24bc8/volumes/kubernetes.io~projected/kube-api-access-ppgng:{mountpoint:/var/lib/kubelet/pods/e588ae83-565a-4711-a7b7-d77b8ae24bc8/volumes/kubernetes.io~projected/kube-api-access-ppgng major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef721fd3-36cc-4fc3-b17c-6ae016daf10c/volumes/kubernetes.io~projected/kube-api-access-x8mg9:{mountpoint:/var/lib/kubelet/pods/ef721fd3-36cc-4fc3-b17c-6ae016daf10c/volumes/kubernetes.io~projected/kube-api-access-x8mg9 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef721fd3-36cc-4fc3-b17c-6ae016daf10c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ef721fd3-36cc-4fc3-b17c-6ae016daf10c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0ed3429-8e54-4f88-b820-751eacadd01a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f0ed3429-8e54-4f88-b820-751eacadd01a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0ed3429-8e54-4f88-b820-751eacadd01a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f0ed3429-8e54-4f88-b820-751eacadd01a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0ed3429-8e54-4f88-b820-751eacadd01a/volumes/kubernetes.io~projected/kube-api-access-cxbrc:{mountpoint:/var/lib/kubelet/pods/f0ed3429-8e54-4f88-b820-751eacadd01a/volumes/kubernetes.io~projected/kube-api-access-cxbrc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2ebae9e-27da-4d11-93c6-acc7657cd4d5/volumes/kubernetes.io~projected/kube-api-access-qc678:{mountpoint:/var/lib/kubelet/pods/f2ebae9e-27da-4d11-93c6-acc7657cd4d5/volumes/kubernetes.io~projected/kube-api-access-qc678 major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2ebae9e-27da-4d11-93c6-acc7657cd4d5/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/f2ebae9e-27da-4d11-93c6-acc7657cd4d5/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2ebae9e-27da-4d11-93c6-acc7657cd4d5/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/f2ebae9e-27da-4d11-93c6-acc7657cd4d5/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fafdd19a-1c68-4429-9519-7da446447921/volumes/kubernetes.io~projected/kube-api-access-6bmns:{mountpoint:/var/lib/kubelet/pods/fafdd19a-1c68-4429-9519-7da446447921/volumes/kubernetes.io~projected/kube-api-access-6bmns major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fafdd19a-1c68-4429-9519-7da446447921/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/fafdd19a-1c68-4429-9519-7da446447921/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fafdd19a-1c68-4429-9519-7da446447921/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/fafdd19a-1c68-4429-9519-7da446447921/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/98783017b7a3ba76f4681f4afad7fcb03f579ad8643d757e2038f3ce2b960b5c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/cdfece662f8eb0743026b311b113535ace8311b7125bb3a5b69e314fe0a4947c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/62ceeb277c5c8ed36f4f02028aa81a1969ae8498d9b49d63b58a2d68bbbb0299/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/4faf081eabfec6c15ab923ab8547a04faf3509efe8d99e23c543e25efdf6ac13/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/69448f7ef0148f6062d555b76beaf676cc2315cf03351c0b80eac834fd517e52/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/12423344e98093bf57bf3b85f15c93ae2b31e2af89bd7df3cb9bc8bc0f6c5ca6/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f84ff8135b8d567b50b05010d25762b8a8fd15499f4ba3d45e969a44ce159ccd/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1bc0b549a633ce27f5c82ce025b5df7d731e69c0b56030e383815f1adb153394/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/be7af2277555a95f71e2303ffd6c7a6ffc9126c5761e9496371fffa2150a1c9c/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/1f9667b48f44c6e4429e13e4d64b87b2da70f4aa4796a0375e33db38ea3201a7/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/84c31c4017dcc2c6815d667d2f3bd1398dcae322cd53d7567e093fbe34d79c1e/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/49ac61a35524cfba55c5af4fd293f90229217fe1b3abb4bcfa7c7c35ae4d172a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/121b3c1d40f5dcf8209ce42b9588820df9310dcfc31b4549f05b21d5d5530e95/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7f9f5c2674c14e6c1600a588238165956264a035e77a43a16abeb3c1bdab76cd/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f7180e4034789ba1d13cc73e088d9664f18d2edeaea108693dc0b061df52d2c6/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/dd6c010e6ef0f6a196a16d3167f3aa30196c20931cd47bc8a3b28cd349478af2/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/aaee7ee43a9cd706c8e424679083acbe76be50bd221abe0176a4d05d97c272a6/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1aa4c9edfba974856ec13cf7a4bc171ee962023bac0ec8a5655a03e1e11e37d9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/215df331dea59da7d97e493b160f389f0c2bca3a88b09fc93f1b863e9820a250/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/683acd5f4bb803d8ca77c69b5c350894105666a5f40e9db29e400cec54a6d83d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8c6fada26a5438f709ff190b202a5884afeb4d362e5f4b4ff3af2d3bf26a4c70/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/3691006afff26b858dfe177e24651c7c272ede935412a77a0bd13a507e36e999/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/7e33f7594fdb27aea4a2ff3aa5fbec19f85191b977043ffd042db8e329fa6aad/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/e085e6a2ee1f3ee6f24c7207071f75c086110ba1e2e1d180310ec9e3f4a574cb/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/9625e98e9ed4d0e8b8104d2904ef51ad76a81233774782614c548600f7196e69/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/d3c7eef9d2ddd57da26398b056ab8ba7fa0656f2eef7d547c8adcd05dd492aca/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/af5abe32ee4d12fccd7545115ac2a274a0f08e4deaf8995d75104cba82e93e95/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/fc9cec61fcfcfb59ec191ee9d6543b266db00f915ddd754c99b26f8480dee04e/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/97a5e8f7c9e53b62c544c1d7e891fa0030906595e0fa660e90a98333db9d73b8/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/d7ff3b578a0ca278692071584b7fb856555cc47f3b5ff726664450602db1dfee/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/3e3da3554a933f768d70028281a90dd4f07e6086a612caa071b4c7093c90e280/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/26ad3a55ec49cd62ca849832177c87dc5f0441ba505e9ed927e305a30c79e5f7/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/0bfc1bc58fddf5b142650b82726c1836307123f750471cd7d4660a737008717d/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/8d13559893e1e23faaedc8841fdaf66089cd1e7285e6567f259835779db8356c/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/e67bfa3bbac764d0bc458b525eff6f1c45057a04775c2121866edee74c19ba53/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/487ef0086b3c816ef73ed6ef1e332de7d1b833654dc9b19ea2f3776e6e88b927/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/98c82bb3d6ac836ce236c2c50ed46e7ea029d7962d7a3dc5e07db90ee2a0dc7c/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/64d8d2f0c1f6c7638eef6fc46d6946524c5b3571afdcbb76770dec09fcbd1d20/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/b51b70735cd830c6b2671df470d6b4ac91c835d3a4276a8f4b51e865708a1b31/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/5736fc26dd7e3f060867bf2dfe9bc1ade4ca647faea04f2428542a0d614c4054/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/6ddbb27238bc4c0a2a79baf5695a7d6ebeebfb209ef1409ff39b5a7aca6bede9/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/371df40f6c8884d5aec637f717f726d361576f21a1f64fb7587a5c4c80313449/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/b83195a09d238c05890d6ff20033754f3af61bc9ec32f6ccfc00317c10a76e6c/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/d3692ce81502c74d5e86e97ba50998dff2b456966b780e860898e1ab91d70d84/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/a7d96fa711956ee313c68d90c6dcd73e001cf789d9b33eb8bc6f354ba1237fd8/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/87307909b74e0b1122bcaeeaefc963b0cfbf0f560c3c18937112cfe04185ba59/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/f9ad77f218c0b31f34080cb63f32d88ae5b4fa46ea044c9ba80d3e15a7ffb976/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/96f335f4061b425e14d78fb1a59672fde3ae2e863405454eec1dc4108e7d6175/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/c12af6ac4a3647ca474e7ca21694d866183d99d9898b1faefb2ff94a00dc3099/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/000154eb0b87b87aae9abaff6187af221da487092c7c793c748c980d8baa7f6e/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/2ac5dc4f985fecc757dca8598029c8a374af26d32d2de405c660869a58a5aa6b/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/dce0b94469d54b311407e64fdb6ac0355f6cc7e89be2f6b0d9a274219c812ada/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/55faa741063b14b82571ad68289464667db2f9201fd04ed6c6772448a0e4a06e/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/525b36c5f069619770c9aabe6a9b1139d6b8672489bb10d2a34f5a536b5218f7/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/d0fdf91a5e6bd42513094fd1e8102939cdb7ad4c49c152b878ac37d3ef21b55b/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/909cfcfc872b16eb728a0cf3e19966f5f7f80dee624b409a77ceffca70f54b8b/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/d2071fa2228c1ad4cecc7b8e17ca2f7bbff5fb8cee402e54b04d39733ccef0af/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/0324441bbf4b69f1aaa047587ed119e4a88018abcd858802a816ef048962789c/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/5c6764c5d3c4a2eb3210a1b86ee6fdb15138b473e2264f00c0c3536ce113aace/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/4f95326cfe28affdb4e5f7e1821db431e3f905353e2e910d1656ac4065cf7775/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/c0a87d2ca2270f3765c38ec57ac2af34a79e45736c08e89b311e5fa184972045/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/33e49b22de3e83bc436504d4de47d415b95a279f331258205ca49e5990b28d6a/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/dbff00623bb5960687ecc9678a0ad034b68b9de4a173b1ca8ec4843ab13a0d8c/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/56f8f1efed7aec9ca661ce34c6dcc25e417332a7503f599318379e1f5acd490a/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/db577ade8bb8c34a6f3a2e72fbf4655861825631189c73505424360d4ee064ca/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/75e7b1cbe716b7f9ced3c46a401a8c15c7c41a9e1ce14715848799acd1cc34b2/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/1b6c21213b53a0cd9a792793257036c425ab26532fac0e68733fce427c1afca6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/6a9de8ecc7e6663d5cbd7b93214e5eb4037a1040af9ec8bd73ba9a9e6a6c620b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/d5f2dfdd1f517d585730879cbbdcdf66b2cef8ac9584d8bc90c021f5cbb9e0ac/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9879ed66cfffd660613448f49403c39b2b1372dab63764028ab44f1ed13e2c4a/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/e3d851b4c8b7040b708fc48a836478681cbada901f51d6625a9074cccac41f77/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8fbe31c963cdd95b425aaff5a03c5e163240bdcda906bfff2d6bb6b15867ad07/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/24f4db86839dbf38358efc6d9c35dc1ffd174cf18f1805047a3267a42899ac12/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 19:52:43.258234 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.257127 5561 manager.go:217] Machine: {Timestamp:2026-11-02 19:52:43.256601438 +0000 UTC m=+0.064385646 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108860 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d99ecaa85c0424ce058bd9a9ddba6 SystemUUID:ec2d99ec-aa85-c042-4ce0-58bd9a9ddba6 BootID:bd7e8f4a-b11a-4480-b779-d12b34fa6dd1 Filesystems:[{Device:/var/lib/kubelet/pods/54ff0392-2388-465d-94b5-c2839f6eac84/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c10c2efe-3e34-4ca7-b972-680cde439609/volumes/kubernetes.io~projected/kube-api-access-4v7m7 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f0ed3429-8e54-4f88-b820-751eacadd01a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a2a3a7bb67cda384d06d04ecde62c7e5e7c574c2712ebc9f2798a5fae4aac8e/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02fc097a077a44cf61b61260e0a9c49039a5fb31363458ad74e8dd86c890b543/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dde184d4c22da9ef2897be99a6e57bd6448f62f35cad908c9e83dadb882b4eb/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/114df931-a2e1-4569-95c9-b7cc0c440079/volumes/kubernetes.io~projected/kube-api-access-j2f4h DeviceMajor:0 DeviceMinor:344 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02867403e1c69adf635a3633653707f6adc03ce1e7eb96c99d50185c71b76e3d/userdata/shm DeviceMajor:0 DeviceMinor:414 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab821fc-d8d9-4ed9-86df-56979b6ac88a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef721fd3-36cc-4fc3-b17c-6ae016daf10c/volumes/kubernetes.io~projected/kube-api-access-x8mg9 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:327 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab5d393663b3d2aefb1aeaa801d18001114cadd1d6df0260dd98f18de4fdc46c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a409cedc-aedc-4130-aad6-4405f1539799/volumes/kubernetes.io~projected/kube-api-access-9x8ws DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0170aa95-6bc5-4107-98f6-7beea4227f4f/volumes/kubernetes.io~projected/kube-api-access-9t966 DeviceMajor:0 DeviceMinor:265 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d60e019-846f-497e-a1fb-1c972295523b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:404 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d10bb7fa-2eb9-4215-b467-c2165c4c76f5/volumes/kubernetes.io~projected/kube-api-access-s4j5s DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31198e0b2a84a17c7d0d20a0ff5f15365a71d4ef25070fea99b96b564632a206/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9/volumes/kubernetes.io~projected/kube-api-access-gnwzr DeviceMajor:0 DeviceMinor:413 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fafdd19a-1c68-4429-9519-7da446447921/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fafdd19a-1c68-4429-9519-7da446447921/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58558a20b85541cdf8b2dbde0ee288393a098121a06a7d15ad31d4ad86139082/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/002e660f-77e7-4384-a7ca-514cc40a4cb6/volumes/kubernetes.io~projected/kube-api-access-mw92p DeviceMajor:0 DeviceMinor:338 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/002e660f-77e7-4384-a7ca-514cc40a4cb6/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:337 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5d86ef2-bb3e-4128-a81e-875fd9f6c527/volumes/kubernetes.io~projected/kube-api-access-xbxrc DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40db643d8f04126b0b7301e1a1e4bf4b530d7c17b75425d0595aee8b64ddbcfb/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0ed3429-8e54-4f88-b820-751eacadd01a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57/volumes/kubernetes.io~projected/kube-api-access-c87tb DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/98798482-f8b5-401e-9aa2-363cda917fba/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:329 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/114df931-a2e1-4569-95c9-b7cc0c440079/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:342 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f2ebae9e-27da-4d11-93c6-acc7657cd4d5/volumes/kubernetes.io~projected/kube-api-access-qc678 DeviceMajor:0 DeviceMinor:353 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6dde0e70-73a7-4198-831e-875401c4c3cd/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:408 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d39bbb7bf19d99d8016b542db8d07a6d84fb326f5c9f5610a1768ccca21a035/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab821fc-d8d9-4ed9-86df-56979b6ac88a/volumes/kubernetes.io~projected/kube-api-access-wwzsj DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0170aa95-6bc5-4107-98f6-7beea4227f4f/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:264 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f28cfd977351c65133951958224c2e26bfcd8673d78cb02f2e20d41e0c6b978/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9957bdb38b1f268e3b8c2ca16ad13609c65ae293836d9f9923f4c5a1e1e0826e/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~projected/kube-api-access-vknpr DeviceMajor:0 DeviceMinor:332 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/114df931-a2e1-4569-95c9-b7cc0c440079/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/459f419345c9e0095001a97e900a384a3394c085d5754d58406123ddc91eef11/userdata/shm DeviceMajor:0 DeviceMinor:409 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3823ef4-9aae-417c-bc9a-a04253589bf5/volumes/kubernetes.io~projected/kube-api-access-fgkrt DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8cd86680e0044abba406a9e8aaa8f180955fdbcb150e85487e7246c730f853f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98798482-f8b5-401e-9aa2-363cda917fba/volumes/kubernetes.io~projected/kube-api-access-bb5kf DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef721fd3-36cc-4fc3-b17c-6ae016daf10c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/114df931-a2e1-4569-95c9-b7cc0c440079/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:343 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f7105b2ce2bca00f516d0fcc551248c1ffada61b655810271ee923b37ea20ad/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9972847e89925e9621b618b0a2bfd6449cd0e69e9cbc206944b2bcbc0cec80c3/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:331 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a409cedc-aedc-4130-aad6-4405f1539799/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/263cb0b0be40e4059ac1a4f5018e966d5ee3bfd6a417af4702b2846d7ed335d8/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99f0054bcd118b2b376e5e18f7804638eb0cf21297f498e7c21d07305461efb7/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5d86ef2-bb3e-4128-a81e-875fd9f6c527/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/092fd489-4d42-46d1-a178-e95204a400e8/volumes/kubernetes.io~projected/kube-api-access-jmg6s DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/744fd9bce60a29357fe1ade982d425ed1b4677f9d68a08754b505689f0a865f2/userdata/shm DeviceMajor:0 DeviceMinor:345 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0ed3429-8e54-4f88-b820-751eacadd01a/volumes/kubernetes.io~projected/kube-api-access-cxbrc DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e/volumes/kubernetes.io~projected/kube-api-access-q46jj DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad4140c0ae0093f04b2e4d9ec3d2214f289e7f173039b984103ae84a8a89c560/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0170aa95-6bc5-4107-98f6-7beea4227f4f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ecaa60df66aea858c5e29edde6204dd9e9ac319b453223e0e3d2100da7b0294/userdata/shm DeviceMajor:0 DeviceMinor:276 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1aa740e4cd6c008dcabbca2e4cdf195848a227921fde726f1ba39f42d5381d92/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5d86ef2-bb3e-4128-a81e-875fd9f6c527/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:268 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57e961e28fea58c3461f0ce6e52c6bac9a40fdfa7581f49eacea6a3fb7adabcd/userdata/shm DeviceMajor:0 DeviceMinor:411 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d60e019-846f-497e-a1fb-1c972295523b/volumes/kubernetes.io~projected/kube-api-access-s4j9w DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:328 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/98798482-f8b5-401e-9aa2-363cda917fba/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/092fd489-4d42-46d1-a178-e95204a400e8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:266 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73f662b539aae6ba3dc20130eaddfea9caf3ca04e7fd2c4d52b6ad2477be4367/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca8189f849c255c459632645fbb1e2159d535964b6fc61524994834c4b1f35df/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fafdd19a-1c68-4429-9519-7da446447921/volumes/kubernetes.io~projected/kube-api-access-6bmns DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e588ae83-565a-4711-a7b7-d77b8ae24bc8/volumes/kubernetes.io~projected/kube-api-access-ppgng DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/092fd489-4d42-46d1-a178-e95204a400e8/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:267 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/367d4910-a962-43b9-a7e6-a58b47636a24/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:330 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f2ebae9e-27da-4d11-93c6-acc7657cd4d5/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:351 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f2ebae9e-27da-4d11-93c6-acc7657cd4d5/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:352 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02867403e1c69ad MacAddress:9e:6a:cd:98:af:e1 Speed:10000 Mtu:8901} {Name:1aa740e4cd6c008 MacAddress:e2:c6:75:d6:d6:02 Speed:10000 Mtu:8901} {Name:1dde184d4c22da9 MacAddress:e6:4d:1c:ad:a3:0a Speed:10000 Mtu:8901} {Name:263cb0b0be40e40 MacAddress:f2:a9:80:75:de:23 Speed:10000 Mtu:8901} {Name:2f28cfd977351c6 MacAddress:fe:1a:1c:95:41:f1 Speed:10000 Mtu:8901} {Name:40db643d8f04126 MacAddress:d6:8b:aa:06:35:41 Speed:10000 Mtu:8901} {Name:459f419345c9e00 MacAddress:32:e7:bc:f6:d8:b3 Speed:10000 Mtu:8901} {Name:57e961e28fea58c MacAddress:da:29:f4:d0:19:74 Speed:10000 Mtu:8901} {Name:6ecaa60df66aea8 MacAddress:2e:c5:1c:ae:01:ae Speed:10000 Mtu:8901} {Name:744fd9bce60a293 MacAddress:b6:60:55:83:3d:4b Speed:10000 Mtu:8901} {Name:9957bdb38b1f268 MacAddress:12:24:c4:98:1c:d5 Speed:10000 Mtu:8901} {Name:99f0054bcd118b2 MacAddress:9a:5e:60:1e:5c:6c Speed:10000 Mtu:8901} {Name:ad4140c0ae0093f MacAddress:e6:ba:9f:af:c8:fb Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:fa:1d:fa:75 Speed:0 Mtu:9001} {Name:br-int MacAddress:5a:5c:8c:50:d8:6a Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:fa:1d:fa:75 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b6:bb:28:29:9e:07 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:7c:6e:23:a3:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 19:52:43.258234 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.257556 5561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 19:52:43.258234 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.257605 5561 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 19:52:43.258234 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.257923 5561 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 19:52:43.258234 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.257953 5561 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-194.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 19:52:43.258234 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.258084 5561 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 19:52:43.258234 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.258092 5561 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 19:52:43.258234 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.258102 5561 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:52:43.258234 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.258121 5561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:52:43.258234 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.258222 5561 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 19:52:43.258682 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.258385 5561 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 19:52:43.258682 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.258487 5561 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 19:52:43.258682 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.258520 5561 kubelet.go:493] "Attempting to sync node with API server" Nov 02 19:52:43.258682 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.258533 5561 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 19:52:43.258682 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.258543 5561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 19:52:43.258682 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.258552 5561 kubelet.go:405] "Adding apiserver pod source" Nov 02 19:52:43.258682 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.258566 5561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 19:52:43.259247 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.259233 5561 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 19:52:43.259247 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.259246 5561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:52:43.259763 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.259742 5561 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 19:52:43.262354 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262339 5561 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 19:52:43.262416 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262358 5561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:52:43.262484 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262474 5561 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 19:52:43.262672 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262662 5561 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 19:52:43.262703 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262691 5561 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 19:52:43.262787 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262779 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 19:52:43.262820 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262792 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 19:52:43.262820 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262798 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 19:52:43.262820 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262804 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 19:52:43.262820 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262810 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 19:52:43.262820 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262816 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 19:52:43.262820 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262822 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 19:52:43.263003 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262828 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 19:52:43.263003 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262835 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 19:52:43.263003 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262840 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 19:52:43.263003 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262849 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 19:52:43.263003 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262862 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 19:52:43.263003 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262907 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 19:52:43.263003 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.262914 5561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 19:52:43.266306 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.266294 5561 server.go:1263] "Started kubelet" Nov 02 19:52:43.266714 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.266375 5561 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 19:52:43.266714 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.266375 5561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 19:52:43.266714 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.266451 5561 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 19:52:43.266902 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.266773 5561 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 19:52:43.266857 ip-10-0-134-194 systemd[1]: Started Kubernetes Kubelet. Nov 02 19:52:43.269511 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.269459 5561 server.go:317] "Adding debug handlers to kubelet server" Nov 02 19:52:43.272771 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.272754 5561 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 19:52:43.272918 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:52:43.272897 5561 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:52:43.272990 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.272947 5561 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 19:52:43.272990 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.272955 5561 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 19:52:43.273703 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.273029 5561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 19:52:43.273703 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.273183 5561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 19:52:43.273703 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.273250 5561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:46:34 +0000 UTC" deadline="2028-08-14 10:43:02.06542315 +0000 UTC" Nov 02 19:52:43.273703 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.273293 5561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15614h50m18.79213328s" Nov 02 19:52:43.274277 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.274262 5561 factory.go:153] Registering CRI-O factory Nov 02 19:52:43.274277 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.274280 5561 factory.go:223] Registration of the crio container factory successfully Nov 02 19:52:43.274420 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.274332 5561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 19:52:43.274420 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.274342 5561 factory.go:55] Registering systemd factory Nov 02 19:52:43.274420 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.274349 5561 factory.go:223] Registration of the systemd container factory successfully Nov 02 19:52:43.274420 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.274365 5561 factory.go:103] Registering Raw factory Nov 02 19:52:43.274420 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.274379 5561 manager.go:1199] Started watching for new ooms in manager Nov 02 19:52:43.274726 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.274711 5561 manager.go:319] Starting recovery of all containers Nov 02 19:52:43.275381 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275339 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="114df931-a2e1-4569-95c9-b7cc0c440079" volumeName="kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-secret-metrics-server-tls" seLinuxMountContext="" Nov 02 19:52:43.275482 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275385 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a409cedc-aedc-4130-aad6-4405f1539799" volumeName="kubernetes.io/configmap/a409cedc-aedc-4130-aad6-4405f1539799-config-volume" seLinuxMountContext="" Nov 02 19:52:43.275482 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275400 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fafdd19a-1c68-4429-9519-7da446447921" volumeName="kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-console-config" seLinuxMountContext="" Nov 02 19:52:43.275482 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275424 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="367d4910-a962-43b9-a7e6-a58b47636a24" volumeName="kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-grpc-tls" seLinuxMountContext="" Nov 02 19:52:43.275482 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275437 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98798482-f8b5-401e-9aa2-363cda917fba" volumeName="kubernetes.io/secret/98798482-f8b5-401e-9aa2-363cda917fba-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 19:52:43.275482 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275449 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fafdd19a-1c68-4429-9519-7da446447921" volumeName="kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-serving-cert" seLinuxMountContext="" Nov 02 19:52:43.275482 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275460 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d60e019-846f-497e-a1fb-1c972295523b" volumeName="kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs" seLinuxMountContext="" Nov 02 19:52:43.275482 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275471 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="002e660f-77e7-4384-a7ca-514cc40a4cb6" volumeName="kubernetes.io/projected/002e660f-77e7-4384-a7ca-514cc40a4cb6-kube-api-access-mw92p" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275485 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="092fd489-4d42-46d1-a178-e95204a400e8" volumeName="kubernetes.io/secret/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-tls" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275499 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c6f9fda-5459-4d5a-8405-7eab0de2cb1e" volumeName="kubernetes.io/configmap/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-serviceca" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275510 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57" volumeName="kubernetes.io/configmap/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-daemon-config" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275522 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5d86ef2-bb3e-4128-a81e-875fd9f6c527" volumeName="kubernetes.io/configmap/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-metrics-client-ca" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275534 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5d86ef2-bb3e-4128-a81e-875fd9f6c527" volumeName="kubernetes.io/empty-dir/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-volume-directive-shadow" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275545 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef721fd3-36cc-4fc3-b17c-6ae016daf10c" volumeName="kubernetes.io/secret/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275560 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0170aa95-6bc5-4107-98f6-7beea4227f4f" volumeName="kubernetes.io/projected/0170aa95-6bc5-4107-98f6-7beea4227f4f-kube-api-access-9t966" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275587 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0170aa95-6bc5-4107-98f6-7beea4227f4f" volumeName="kubernetes.io/secret/0170aa95-6bc5-4107-98f6-7beea4227f4f-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275598 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="114df931-a2e1-4569-95c9-b7cc0c440079" volumeName="kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275611 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef721fd3-36cc-4fc3-b17c-6ae016daf10c" volumeName="kubernetes.io/empty-dir/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-data-volume" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275623 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0ed3429-8e54-4f88-b820-751eacadd01a" volumeName="kubernetes.io/empty-dir/f0ed3429-8e54-4f88-b820-751eacadd01a-tmp" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275634 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0ed3429-8e54-4f88-b820-751eacadd01a" volumeName="kubernetes.io/empty-dir/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-tuned" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275646 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" volumeName="kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-oauth-serving-cert" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275672 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" volumeName="kubernetes.io/empty-dir/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-tmp" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275683 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="092fd489-4d42-46d1-a178-e95204a400e8" volumeName="kubernetes.io/configmap/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275696 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="114df931-a2e1-4569-95c9-b7cc0c440079" volumeName="kubernetes.io/empty-dir/114df931-a2e1-4569-95c9-b7cc0c440079-audit-log" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275709 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e588ae83-565a-4711-a7b7-d77b8ae24bc8" volumeName="kubernetes.io/empty-dir/e588ae83-565a-4711-a7b7-d77b8ae24bc8-tmp-dir" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275720 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" volumeName="kubernetes.io/projected/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-kube-api-access-qc678" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275732 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" volumeName="kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-serving-cert" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275746 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="092fd489-4d42-46d1-a178-e95204a400e8" volumeName="kubernetes.io/empty-dir/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-textfile" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275776 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="367d4910-a962-43b9-a7e6-a58b47636a24" volumeName="kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275789 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98798482-f8b5-401e-9aa2-363cda917fba" volumeName="kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-ovnkube-config" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275801 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98798482-f8b5-401e-9aa2-363cda917fba" volumeName="kubernetes.io/projected/98798482-f8b5-401e-9aa2-363cda917fba-kube-api-access-bb5kf" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275812 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e588ae83-565a-4711-a7b7-d77b8ae24bc8" volumeName="kubernetes.io/projected/e588ae83-565a-4711-a7b7-d77b8ae24bc8-kube-api-access-ppgng" seLinuxMountContext="" Nov 02 19:52:43.275815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275824 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef721fd3-36cc-4fc3-b17c-6ae016daf10c" volumeName="kubernetes.io/configmap/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 19:52:43.276948 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275837 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fafdd19a-1c68-4429-9519-7da446447921" volumeName="kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-service-ca" seLinuxMountContext="" Nov 02 19:52:43.276948 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275850 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a409cedc-aedc-4130-aad6-4405f1539799" volumeName="kubernetes.io/projected/a409cedc-aedc-4130-aad6-4405f1539799-kube-api-access-9x8ws" seLinuxMountContext="" Nov 02 19:52:43.277920 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.275862 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="002e660f-77e7-4384-a7ca-514cc40a4cb6" volumeName="kubernetes.io/secret/002e660f-77e7-4384-a7ca-514cc40a4cb6-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 19:52:43.277996 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.277930 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="367d4910-a962-43b9-a7e6-a58b47636a24" volumeName="kubernetes.io/empty-dir/367d4910-a962-43b9-a7e6-a58b47636a24-tmp" seLinuxMountContext="" Nov 02 19:52:43.277996 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.277947 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="367d4910-a962-43b9-a7e6-a58b47636a24" volumeName="kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278182 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="367d4910-a962-43b9-a7e6-a58b47636a24" volumeName="kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-tls" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278315 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98798482-f8b5-401e-9aa2-363cda917fba" volumeName="kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-ovnkube-script-lib" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278335 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3823ef4-9aae-417c-bc9a-a04253589bf5" volumeName="kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278381 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a409cedc-aedc-4130-aad6-4405f1539799" volumeName="kubernetes.io/empty-dir/a409cedc-aedc-4130-aad6-4405f1539799-tmp-dir" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278395 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a409cedc-aedc-4130-aad6-4405f1539799" volumeName="kubernetes.io/secret/a409cedc-aedc-4130-aad6-4405f1539799-metrics-tls" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278479 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c10c2efe-3e34-4ca7-b972-680cde439609" volumeName="kubernetes.io/configmap/c10c2efe-3e34-4ca7-b972-680cde439609-iptables-alerter-script" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278517 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="092fd489-4d42-46d1-a178-e95204a400e8" volumeName="kubernetes.io/configmap/092fd489-4d42-46d1-a178-e95204a400e8-metrics-client-ca" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278570 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="367d4910-a962-43b9-a7e6-a58b47636a24" volumeName="kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278589 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10bb7fa-2eb9-4215-b467-c2165c4c76f5" volumeName="kubernetes.io/projected/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-kube-api-access-s4j5s" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278601 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fafdd19a-1c68-4429-9519-7da446447921" volumeName="kubernetes.io/empty-dir/fafdd19a-1c68-4429-9519-7da446447921-tmp" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278653 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="114df931-a2e1-4569-95c9-b7cc0c440079" volumeName="kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-client-ca-bundle" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278676 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6dde0e70-73a7-4198-831e-875401c4c3cd" volumeName="kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278688 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fafdd19a-1c68-4429-9519-7da446447921" volumeName="kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-oauth-serving-cert" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278722 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0170aa95-6bc5-4107-98f6-7beea4227f4f" volumeName="kubernetes.io/configmap/0170aa95-6bc5-4107-98f6-7beea4227f4f-metrics-client-ca" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278761 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="114df931-a2e1-4569-95c9-b7cc0c440079" volumeName="kubernetes.io/configmap/114df931-a2e1-4569-95c9-b7cc0c440079-metrics-server-audit-profiles" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278779 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ff0392-2388-465d-94b5-c2839f6eac84" volumeName="kubernetes.io/secret/54ff0392-2388-465d-94b5-c2839f6eac84-agent-certs" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278805 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57" volumeName="kubernetes.io/configmap/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-cni-binary-copy" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278818 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98798482-f8b5-401e-9aa2-363cda917fba" volumeName="kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-env-overrides" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278899 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2ff0bc2-a4cf-45f3-b870-2526d024b6c9" volumeName="kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278935 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5d86ef2-bb3e-4128-a81e-875fd9f6c527" volumeName="kubernetes.io/secret/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.278947 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" volumeName="kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-config" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279089 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0170aa95-6bc5-4107-98f6-7beea4227f4f" volumeName="kubernetes.io/secret/0170aa95-6bc5-4107-98f6-7beea4227f4f-openshift-state-metrics-tls" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279126 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fafdd19a-1c68-4429-9519-7da446447921" volumeName="kubernetes.io/projected/fafdd19a-1c68-4429-9519-7da446447921-kube-api-access-6bmns" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279140 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fafdd19a-1c68-4429-9519-7da446447921" volumeName="kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-oauth-config" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279157 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="092fd489-4d42-46d1-a178-e95204a400e8" volumeName="kubernetes.io/projected/092fd489-4d42-46d1-a178-e95204a400e8-kube-api-access-jmg6s" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279177 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="367d4910-a962-43b9-a7e6-a58b47636a24" volumeName="kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279190 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3823ef4-9aae-417c-bc9a-a04253589bf5" volumeName="kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-cni-binary-copy" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279215 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3823ef4-9aae-417c-bc9a-a04253589bf5" volumeName="kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279228 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c10c2efe-3e34-4ca7-b972-680cde439609" volumeName="kubernetes.io/projected/c10c2efe-3e34-4ca7-b972-680cde439609-kube-api-access-4v7m7" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279242 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef721fd3-36cc-4fc3-b17c-6ae016daf10c" volumeName="kubernetes.io/projected/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-kube-api-access-x8mg9" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279279 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0ed3429-8e54-4f88-b820-751eacadd01a" volumeName="kubernetes.io/projected/f0ed3429-8e54-4f88-b820-751eacadd01a-kube-api-access-cxbrc" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279294 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" volumeName="kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-oauth-config" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279312 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="114df931-a2e1-4569-95c9-b7cc0c440079" volumeName="kubernetes.io/projected/114df931-a2e1-4569-95c9-b7cc0c440079-kube-api-access-j2f4h" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279327 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab821fc-d8d9-4ed9-86df-56979b6ac88a" volumeName="kubernetes.io/projected/2ab821fc-d8d9-4ed9-86df-56979b6ac88a-kube-api-access-wwzsj" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279341 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c6f9fda-5459-4d5a-8405-7eab0de2cb1e" volumeName="kubernetes.io/projected/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-kube-api-access-q46jj" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279360 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5d86ef2-bb3e-4128-a81e-875fd9f6c527" volumeName="kubernetes.io/configmap/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279386 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5d86ef2-bb3e-4128-a81e-875fd9f6c527" volumeName="kubernetes.io/projected/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-api-access-xbxrc" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279410 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" volumeName="kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-trusted-ca-bundle" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279424 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="367d4910-a962-43b9-a7e6-a58b47636a24" volumeName="kubernetes.io/projected/367d4910-a962-43b9-a7e6-a58b47636a24-kube-api-access-vknpr" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279440 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab821fc-d8d9-4ed9-86df-56979b6ac88a" volumeName="kubernetes.io/secret/2ab821fc-d8d9-4ed9-86df-56979b6ac88a-cert" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279459 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="367d4910-a962-43b9-a7e6-a58b47636a24" volumeName="kubernetes.io/configmap/367d4910-a962-43b9-a7e6-a58b47636a24-metrics-client-ca" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279476 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ff0392-2388-465d-94b5-c2839f6eac84" volumeName="kubernetes.io/configmap/54ff0392-2388-465d-94b5-c2839f6eac84-konnectivity-ca" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279493 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" volumeName="kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-service-ca" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279508 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="092fd489-4d42-46d1-a178-e95204a400e8" volumeName="kubernetes.io/secret/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279523 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57" volumeName="kubernetes.io/projected/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-kube-api-access-c87tb" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279550 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3823ef4-9aae-417c-bc9a-a04253589bf5" volumeName="kubernetes.io/projected/a3823ef4-9aae-417c-bc9a-a04253589bf5-kube-api-access-fgkrt" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279566 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5d86ef2-bb3e-4128-a81e-875fd9f6c527" volumeName="kubernetes.io/secret/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279586 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d60e019-846f-497e-a1fb-1c972295523b" volumeName="kubernetes.io/projected/0d60e019-846f-497e-a1fb-1c972295523b-kube-api-access-s4j9w" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279600 5561 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="114df931-a2e1-4569-95c9-b7cc0c440079" volumeName="kubernetes.io/configmap/114df931-a2e1-4569-95c9-b7cc0c440079-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279611 5561 reconstruct.go:97] "Volume reconstruction finished" Nov 02 19:52:43.281393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.279621 5561 reconciler.go:29] "Reconciler: start to sync state" Nov 02 19:52:43.304329 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.304273 5561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 19:52:43.305482 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.305459 5561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 19:52:43.305482 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.305481 5561 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 19:52:43.305629 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.305502 5561 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 19:52:43.305629 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:52:43.305553 5561 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 19:52:43.317524 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.317482 5561 generic.go:356] "Generic (PLEG): container finished" podID="770dce62893bdbeaf2564d856d57a2e6" containerID="1564ef31fa65a6b0d086e94f8f790e3068003943ea75f3b122948ab5cee90ca1" exitCode=0 Nov 02 19:52:43.353905 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.353860 5561 generic.go:356] "Generic (PLEG): container finished" podID="a3823ef4-9aae-417c-bc9a-a04253589bf5" containerID="2138975dc1e7fc6a2dddf3dab1f72a75c315b2cac5c251f59294947da4a9938f" exitCode=0 Nov 02 19:52:43.353905 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.353904 5561 generic.go:356] "Generic (PLEG): container finished" podID="a3823ef4-9aae-417c-bc9a-a04253589bf5" containerID="cf1edd4a5e93def852d6e5c0dac6521d4d44e1b1a5a6961524a8cdcaafc6fa9b" exitCode=0 Nov 02 19:52:43.354049 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.353914 5561 generic.go:356] "Generic (PLEG): container finished" podID="a3823ef4-9aae-417c-bc9a-a04253589bf5" containerID="b5a2a7623eb0f792f49497ac05e8b5df4a280676b9945a463842b9a50ab6a586" exitCode=0 Nov 02 19:52:43.354049 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.353923 5561 generic.go:356] "Generic (PLEG): container finished" podID="a3823ef4-9aae-417c-bc9a-a04253589bf5" containerID="90d19e77e182fac56eb0fac99f45202dc496aeba03742adfa1d0eab74bdb634c" exitCode=0 Nov 02 19:52:43.354049 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.353932 5561 generic.go:356] "Generic (PLEG): container finished" podID="a3823ef4-9aae-417c-bc9a-a04253589bf5" containerID="92d2b3f3f0e077cbd8854ef006b98009e3e9b1063e1f5e953b7b49e9f5c3df0b" exitCode=0 Nov 02 19:52:43.354049 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.353940 5561 generic.go:356] "Generic (PLEG): container finished" podID="a3823ef4-9aae-417c-bc9a-a04253589bf5" containerID="54bac71d92eb7cfc5e1ef6a38b686dc913dfffa8cd3623a45e46a001564cbd3f" exitCode=0 Nov 02 19:52:43.360296 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.360270 5561 generic.go:356] "Generic (PLEG): container finished" podID="092fd489-4d42-46d1-a178-e95204a400e8" containerID="25fc319639bab50691dea2ed127def0d71c890e4cb32555b9e014181079b1da8" exitCode=0 Nov 02 19:52:43.373009 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:52:43.372947 5561 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:52:43.379974 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.379954 5561 manager.go:324] Recovery completed Nov 02 19:52:43.399924 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.399904 5561 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:52:43.402132 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.402110 5561 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:52:43.402218 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.402144 5561 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:52:43.402218 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.402160 5561 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:52:43.404886 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.404861 5561 cpu_manager.go:226] "Starting" policy="none" Nov 02 19:52:43.404886 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.404885 5561 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 19:52:43.404886 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.404888 5561 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:52:43.405027 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.404905 5561 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 19:52:43.405089 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405058 5561 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 19:52:43.405120 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405078 5561 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 19:52:43.405120 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405097 5561 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 19:52:43.405120 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405102 5561 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 19:52:43.405120 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405108 5561 policy_none.go:50] "Start" Nov 02 19:52:43.405120 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405117 5561 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 19:52:43.405260 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405127 5561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 19:52:43.405335 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405262 5561 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 19:52:43.405335 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405269 5561 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 19:52:43.405335 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405275 5561 policy_none.go:44] "Start" Nov 02 19:52:43.405633 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:52:43.405615 5561 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 19:52:43.405739 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405716 5561 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:52:43.405803 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405755 5561 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:52:43.405803 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.405767 5561 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:52:43.409762 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.409736 5561 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 19:52:43.412566 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.412554 5561 manager.go:341] "Starting Device Plugin manager" Nov 02 19:52:43.412649 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:52:43.412579 5561 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 19:52:43.412649 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.412591 5561 server.go:87] "Starting device plugin registration server" Nov 02 19:52:43.412911 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.412895 5561 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 19:52:43.412971 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.412915 5561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 19:52:43.413022 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.412978 5561 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 19:52:43.413157 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.413123 5561 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 19:52:43.413252 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.413173 5561 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 19:52:43.413252 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.413180 5561 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 19:52:43.413338 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.413227 5561 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:52:43.413151713Z","UUID":"4cc5b697-9cf3-4535-aa63-b64114b21c7c","Handler":null,"Name":"","Endpoint":""} Nov 02 19:52:43.415738 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:52:43.415711 5561 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 19:52:43.415821 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:52:43.415765 5561 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-194.us-west-2.compute.internal\" not found" Nov 02 19:52:43.416052 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.416035 5561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 19:52:43.416118 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.416063 5561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 19:52:43.459257 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.459231 5561 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:52:43.459345 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.459308 5561 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 19:52:43.459891 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.459859 5561 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:52:43.506221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.506194 5561 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:52:43.513695 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.513681 5561 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:52:43.514615 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.514597 5561 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:52:43.514680 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.514630 5561 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:52:43.514680 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.514641 5561 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:52:43.514768 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.514684 5561 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.554100 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.554074 5561 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:52:43.606628 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.606582 5561 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal"] Nov 02 19:52:43.607241 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.607200 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" event={"ID":"770dce62893bdbeaf2564d856d57a2e6","Type":"ContainerStarted","Data":"47811b61c36d347c19afca864ac647e6932e3b7032cff1995683c472381eed2f"} Nov 02 19:52:43.607322 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.607256 5561 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.607322 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.607261 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" event={"ID":"770dce62893bdbeaf2564d856d57a2e6","Type":"ContainerDied","Data":"1564ef31fa65a6b0d086e94f8f790e3068003943ea75f3b122948ab5cee90ca1"} Nov 02 19:52:43.607322 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.607286 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" event={"ID":"770dce62893bdbeaf2564d856d57a2e6","Type":"ContainerStarted","Data":"73f662b539aae6ba3dc20130eaddfea9caf3ca04e7fd2c4d52b6ad2477be4367"} Nov 02 19:52:43.607322 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.607312 5561 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.607507 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.607333 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" event={"ID":"0470b607b1bfee936d3bbd33e3118f8d","Type":"ContainerStarted","Data":"486428f7f3853e4f4f0d09ad2b3de9388e4dbe5cf19de02dd95b960aadf6f6a3"} Nov 02 19:52:43.607507 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.607354 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" event={"ID":"0470b607b1bfee936d3bbd33e3118f8d","Type":"ContainerStarted","Data":"02fc097a077a44cf61b61260e0a9c49039a5fb31363458ad74e8dd86c890b543"} Nov 02 19:52:43.656945 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.656919 5561 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.657083 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.656990 5561 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.664764 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.664744 5561 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:52:43.664898 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:52:43.664800 5561 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.665236 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.665220 5561 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:52:43.665273 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:52:43.665258 5561 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.761408 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.761377 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/770dce62893bdbeaf2564d856d57a2e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"770dce62893bdbeaf2564d856d57a2e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.761408 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.761407 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/770dce62893bdbeaf2564d856d57a2e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"770dce62893bdbeaf2564d856d57a2e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.761602 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.761427 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0470b607b1bfee936d3bbd33e3118f8d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"0470b607b1bfee936d3bbd33e3118f8d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.862505 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.862476 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/770dce62893bdbeaf2564d856d57a2e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"770dce62893bdbeaf2564d856d57a2e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.862671 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.862512 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0470b607b1bfee936d3bbd33e3118f8d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"0470b607b1bfee936d3bbd33e3118f8d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.862671 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.862533 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/770dce62893bdbeaf2564d856d57a2e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"770dce62893bdbeaf2564d856d57a2e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.862671 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.862579 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/770dce62893bdbeaf2564d856d57a2e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"770dce62893bdbeaf2564d856d57a2e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.862671 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.862603 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0470b607b1bfee936d3bbd33e3118f8d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"0470b607b1bfee936d3bbd33e3118f8d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:43.862671 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:43.862578 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/770dce62893bdbeaf2564d856d57a2e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" (UID: \"770dce62893bdbeaf2564d856d57a2e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:44.259780 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.259741 5561 apiserver.go:52] "Watching apiserver" Nov 02 19:52:44.358568 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.358538 5561 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 19:52:44.359148 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.359122 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal","openshift-image-registry/node-ca-dsct6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk","openshift-ingress-canary/ingress-canary-92vwp","openshift-insights/insights-runtime-extractor-4b5v5","kube-system/konnectivity-agent-r9jxd","openshift-multus/multus-z2vn7","openshift-dns/node-resolver-9rglx","openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5","kube-system/global-pull-secret-syncer-4bpmh","openshift-cluster-node-tuning-operator/tuned-7m9qt","openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal","openshift-dns/dns-default-jqtjb","openshift-monitoring/metrics-server-796b7c8599-d2vtr","openshift-network-operator/iptables-alerter-nqhf8","openshift-network-diagnostics/network-check-target-m2tvk","openshift-ovn-kubernetes/ovnkube-node-r6lg7","openshift-console/console-67c8cb984d-5gwb9","openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9","openshift-monitoring/thanos-querier-7868584944-ktpmj","openshift-monitoring/node-exporter-287j2","openshift-multus/multus-additional-cni-plugins-7svjs","openshift-multus/network-metrics-daemon-r67nn","openshift-console/console-8478dfbff8-ldrtw"] Nov 02 19:52:44.367528 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.367501 5561 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:44.367673 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.367582 5561 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:44.374864 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.374842 5561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 19:52:44.413250 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.413222 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 19:52:44.414272 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.414254 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:52:44.414352 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.414328 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 19:52:44.414617 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.414605 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 19:52:44.414793 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.414769 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 19:52:44.414915 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.414848 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 19:52:44.414915 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.414856 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 19:52:44.415013 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.414918 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 19:52:44.415126 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.415111 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 19:52:44.415271 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.415249 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 19:52:44.415271 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.415251 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 19:52:44.417020 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.416990 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 19:52:44.417335 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417314 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 19:52:44.417504 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417491 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:52:44.417567 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417537 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 19:52:44.417567 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417559 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 19:52:44.417664 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417578 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 19:52:44.417664 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417579 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 19:52:44.417664 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417597 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:52:44.417664 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417597 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 19:52:44.417664 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417579 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 19:52:44.417664 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417615 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 19:52:44.417926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417682 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 19:52:44.417926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417689 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 19:52:44.417926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417711 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 19:52:44.417926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.417730 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:52:44.418218 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.418205 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 19:52:44.425738 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.425709 5561 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:52:44.425814 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:52:44.425768 5561 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:44.426089 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.426074 5561 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:52:44.426149 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:52:44.426128 5561 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-194.us-west-2.compute.internal" Nov 02 19:52:44.449335 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.449314 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ltmn5\"" Nov 02 19:52:44.449949 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.449927 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:52:44.449949 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.449942 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4w4l2\"" Nov 02 19:52:44.450074 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.449929 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 19:52:44.450325 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.450292 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sk47\"" Nov 02 19:52:44.450426 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.450345 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 19:52:44.450513 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.450499 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dbl8w\"" Nov 02 19:52:44.450586 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.450524 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gf4xf\"" Nov 02 19:52:44.450724 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.450700 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f8f9p\"" Nov 02 19:52:44.450777 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.450743 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvmjq\"" Nov 02 19:52:44.450831 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.450818 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 19:52:44.450996 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.450980 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:52:44.451112 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.451098 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4hfb\"" Nov 02 19:52:44.451173 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.451106 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlz48\"" Nov 02 19:52:44.451261 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.451246 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 19:52:44.451355 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.451341 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-shcvr\"" Nov 02 19:52:44.451422 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.451413 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qqgmv\"" Nov 02 19:52:44.451508 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.451497 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vl78h\"" Nov 02 19:52:44.451585 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.451574 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8m7zt\"" Nov 02 19:52:44.467168 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467148 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwzsj\" (UniqueName: \"kubernetes.io/projected/2ab821fc-d8d9-4ed9-86df-56979b6ac88a-kube-api-access-wwzsj\") pod \"ingress-canary-92vwp\" (UID: \"2ab821fc-d8d9-4ed9-86df-56979b6ac88a\") " pod="openshift-ingress-canary/ingress-canary-92vwp" Nov 02 19:52:44.467258 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467176 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-cni-bin\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.467258 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467193 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-cni-multus\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.467258 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467208 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-multus-certs\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.467258 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467226 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0170aa95-6bc5-4107-98f6-7beea4227f4f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:44.467431 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467271 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.467431 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467387 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppgng\" (UniqueName: \"kubernetes.io/projected/e588ae83-565a-4711-a7b7-d77b8ae24bc8-kube-api-access-ppgng\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:52:44.467431 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467421 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-systemd-units\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.467563 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467437 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-systemd\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.467563 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467452 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-node-log\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.467563 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467492 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-oauth-serving-cert\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.467563 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467532 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-textfile\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.467737 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467563 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.467737 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467587 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-cni-bin\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.467737 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467601 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-textfile\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.467737 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467632 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-env-overrides\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.467737 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467669 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:44.467737 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467693 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46jj\" (UniqueName: \"kubernetes.io/projected/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-kube-api-access-q46jj\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:52:44.467737 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467709 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-run-netns\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.467737 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467724 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-config\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467747 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a409cedc-aedc-4130-aad6-4405f1539799-config-volume\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467797 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467819 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467838 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-slash\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467838 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-env-overrides\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467860 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467931 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-modprobe-d\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467933 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a409cedc-aedc-4130-aad6-4405f1539799-config-volume\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.467951 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468012 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw92p\" (UniqueName: \"kubernetes.io/projected/002e660f-77e7-4384-a7ca-514cc40a4cb6-kube-api-access-mw92p\") pod \"monitoring-plugin-77f4f76594-6qrl5\" (UID: \"002e660f-77e7-4384-a7ca-514cc40a4cb6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468038 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-cnibin\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468073 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-sys\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468087 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-root\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468086 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d60e019-846f-497e-a1fb-1c972295523b-metrics-certs\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468117 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-run\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.468131 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468137 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-client-ca-bundle\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468171 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468214 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmns\" (UniqueName: \"kubernetes.io/projected/fafdd19a-1c68-4429-9519-7da446447921-kube-api-access-6bmns\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468267 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-socket-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468282 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e588ae83-565a-4711-a7b7-d77b8ae24bc8-hosts-file\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468300 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-tls\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468346 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/367d4910-a962-43b9-a7e6-a58b47636a24-metrics-client-ca\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468451 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-secret-metrics-server-tls\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468509 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-tmp\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468535 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-tls\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468552 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6dde0e70-73a7-4198-831e-875401c4c3cd-dbus\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468569 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-console-config\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468583 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468652 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-secret-metrics-server-client-certs\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468675 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-tmp\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468678 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-systemd\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468713 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a409cedc-aedc-4130-aad6-4405f1539799-metrics-tls\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:44.468758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468758 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbxrc\" (UniqueName: \"kubernetes.io/projected/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-api-access-xbxrc\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468798 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c10c2efe-3e34-4ca7-b972-680cde439609-iptables-alerter-script\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468815 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-kubelet\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468844 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-serving-cert\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468867 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysctl-d\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468951 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.468960 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a409cedc-aedc-4130-aad6-4405f1539799-metrics-tls\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469023 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c10c2efe-3e34-4ca7-b972-680cde439609-iptables-alerter-script\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469039 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-lib-modules\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469064 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a409cedc-aedc-4130-aad6-4405f1539799-tmp-dir\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469079 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-host\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469097 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6dde0e70-73a7-4198-831e-875401c4c3cd-kubelet-config\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469159 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a409cedc-aedc-4130-aad6-4405f1539799-tmp-dir\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469189 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-ovnkube-script-lib\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469222 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/092fd489-4d42-46d1-a178-e95204a400e8-metrics-client-ca\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469240 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469274 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-wtmp\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469310 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469336 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-etc-kubernetes\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469374 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-log-socket\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469386 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-cni-binary-copy\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469428 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-cni-netd\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469438 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-ovnkube-script-lib\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469452 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-kubernetes\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.469473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469478 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/114df931-a2e1-4569-95c9-b7cc0c440079-audit-log\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469512 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-socket-dir-parent\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469520 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469547 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/114df931-a2e1-4569-95c9-b7cc0c440079-audit-log\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469579 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-daemon-config\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469608 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87tb\" (UniqueName: \"kubernetes.io/projected/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-kube-api-access-c87tb\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469680 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t966\" (UniqueName: \"kubernetes.io/projected/0170aa95-6bc5-4107-98f6-7beea4227f4f-kube-api-access-9t966\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469716 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-grpc-tls\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469753 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469771 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7m7\" (UniqueName: \"kubernetes.io/projected/c10c2efe-3e34-4ca7-b972-680cde439609-kube-api-access-4v7m7\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469785 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-trusted-ca-bundle\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469787 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-daemon-config\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469800 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-hostroot\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469851 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-device-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469898 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-registration-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469927 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/114df931-a2e1-4569-95c9-b7cc0c440079-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.469979 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0ed3429-8e54-4f88-b820-751eacadd01a-tmp\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470019 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbrc\" (UniqueName: \"kubernetes.io/projected/f0ed3429-8e54-4f88-b820-751eacadd01a-kube-api-access-cxbrc\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470043 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470085 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4j5s\" (UniqueName: \"kubernetes.io/projected/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-kube-api-access-s4j5s\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470066 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0ed3429-8e54-4f88-b820-751eacadd01a-tmp\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470108 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-system-cni-dir\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470127 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-serviceca\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470143 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-oauth-config\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470166 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-netns\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470195 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysconfig\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470278 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54ff0392-2388-465d-94b5-c2839f6eac84-agent-certs\") pod \"konnectivity-agent-r9jxd\" (UID: \"54ff0392-2388-465d-94b5-c2839f6eac84\") " pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470325 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-service-ca\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470418 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:44.470466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470438 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-ovnkube-config\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470530 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0170aa95-6bc5-4107-98f6-7beea4227f4f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470557 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vknpr\" (UniqueName: \"kubernetes.io/projected/367d4910-a962-43b9-a7e6-a58b47636a24-kube-api-access-vknpr\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470555 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54ff0392-2388-465d-94b5-c2839f6eac84-agent-certs\") pod \"konnectivity-agent-r9jxd\" (UID: \"54ff0392-2388-465d-94b5-c2839f6eac84\") " pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470587 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-host\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470628 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-serving-cert\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470646 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98798482-f8b5-401e-9aa2-363cda917fba-ovnkube-config\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470686 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-etc-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470733 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-oauth-config\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470775 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-tuned\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470802 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-crio-socket\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470836 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/002e660f-77e7-4384-a7ca-514cc40a4cb6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6qrl5\" (UID: \"002e660f-77e7-4384-a7ca-514cc40a4cb6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470868 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.470947 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-tuned\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471000 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-serviceca\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471024 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2f4h\" (UniqueName: \"kubernetes.io/projected/114df931-a2e1-4569-95c9-b7cc0c440079-kube-api-access-j2f4h\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471058 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471083 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-k8s-cni-cncf-io\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471108 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-var-lib-kubelet\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471136 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e588ae83-565a-4711-a7b7-d77b8ae24bc8-tmp-dir\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471166 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-ovn\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471188 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e588ae83-565a-4711-a7b7-d77b8ae24bc8-tmp-dir\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471232 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/114df931-a2e1-4569-95c9-b7cc0c440079-metrics-server-audit-profiles\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471268 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysctl-conf\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471297 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-oauth-serving-cert\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471325 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471362 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0170aa95-6bc5-4107-98f6-7beea4227f4f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471423 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-system-cni-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471447 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471464 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-cni-binary-copy\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471484 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-kubelet\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.471501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471501 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-conf-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471539 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54ff0392-2388-465d-94b5-c2839f6eac84-konnectivity-ca\") pod \"konnectivity-agent-r9jxd\" (UID: \"54ff0392-2388-465d-94b5-c2839f6eac84\") " pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471556 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafdd19a-1c68-4429-9519-7da446447921-tmp\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471573 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471594 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-var-lib-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471619 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471646 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471647 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafdd19a-1c68-4429-9519-7da446447921-tmp\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471688 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367d4910-a962-43b9-a7e6-a58b47636a24-tmp\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471692 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54ff0392-2388-465d-94b5-c2839f6eac84-konnectivity-ca\") pod \"konnectivity-agent-r9jxd\" (UID: \"54ff0392-2388-465d-94b5-c2839f6eac84\") " pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471741 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-cni-binary-copy\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471776 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367d4910-a962-43b9-a7e6-a58b47636a24-tmp\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471817 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc678\" (UniqueName: \"kubernetes.io/projected/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-kube-api-access-qc678\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471836 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471859 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8ws\" (UniqueName: \"kubernetes.io/projected/a409cedc-aedc-4130-aad6-4405f1539799-kube-api-access-9x8ws\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471892 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-service-ca\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471913 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgkrt\" (UniqueName: \"kubernetes.io/projected/a3823ef4-9aae-417c-bc9a-a04253589bf5-kube-api-access-fgkrt\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471930 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-cni-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471949 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-data-volume\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.471975 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ab821fc-d8d9-4ed9-86df-56979b6ac88a-cert\") pod \"ingress-canary-92vwp\" (UID: \"2ab821fc-d8d9-4ed9-86df-56979b6ac88a\") " pod="openshift-ingress-canary/ingress-canary-92vwp" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472017 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472046 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c10c2efe-3e34-4ca7-b972-680cde439609-host-slash\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472061 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-sys-fs\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472066 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-data-volume\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472063 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a3823ef4-9aae-417c-bc9a-a04253589bf5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472109 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb5kf\" (UniqueName: \"kubernetes.io/projected/98798482-f8b5-401e-9aa2-363cda917fba-kube-api-access-bb5kf\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472136 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-cnibin\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472155 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwzr\" (UniqueName: \"kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr\") pod \"network-check-target-m2tvk\" (UID: \"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9\") " pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472208 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98798482-f8b5-401e-9aa2-363cda917fba-ovn-node-metrics-cert\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472237 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472262 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmg6s\" (UniqueName: \"kubernetes.io/projected/092fd489-4d42-46d1-a178-e95204a400e8-kube-api-access-jmg6s\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472278 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4j9w\" (UniqueName: \"kubernetes.io/projected/0d60e019-846f-497e-a1fb-1c972295523b-kube-api-access-s4j9w\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472294 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-sys\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472319 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8mg9\" (UniqueName: \"kubernetes.io/projected/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-kube-api-access-x8mg9\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472411 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dde0e70-73a7-4198-831e-875401c4c3cd-original-pull-secret\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:44.472667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.472426 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98798482-f8b5-401e-9aa2-363cda917fba-ovn-node-metrics-cert\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.513288 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513229 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 19:52:44.513407 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513336 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 19:52:44.513407 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513357 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 19:52:44.513577 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513413 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 19:52:44.513577 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513427 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 19:52:44.513577 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513527 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 19:52:44.513577 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513569 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 19:52:44.513705 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513576 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 19:52:44.513705 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513625 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 19:52:44.513705 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513691 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 19:52:44.513819 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513774 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 19:52:44.513819 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513791 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 19:52:44.513819 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513799 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 19:52:44.513975 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513957 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 19:52:44.514017 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.513979 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 19:52:44.514293 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.514282 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 19:52:44.514332 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.514289 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 19:52:44.514720 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.514708 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 19:52:44.517816 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.517801 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-oauth-serving-cert\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.518040 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.518025 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-config\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.518803 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.518786 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-console-config\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.518908 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.518895 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/367d4910-a962-43b9-a7e6-a58b47636a24-metrics-client-ca\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.519516 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.519499 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/092fd489-4d42-46d1-a178-e95204a400e8-metrics-client-ca\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.520096 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.520072 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppgng\" (UniqueName: \"kubernetes.io/projected/e588ae83-565a-4711-a7b7-d77b8ae24bc8-kube-api-access-ppgng\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:52:44.520271 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.520254 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46jj\" (UniqueName: \"kubernetes.io/projected/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-kube-api-access-q46jj\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:52:44.520343 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.520279 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/114df931-a2e1-4569-95c9-b7cc0c440079-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.520343 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.520312 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwzsj\" (UniqueName: \"kubernetes.io/projected/2ab821fc-d8d9-4ed9-86df-56979b6ac88a-kube-api-access-wwzsj\") pod \"ingress-canary-92vwp\" (UID: \"2ab821fc-d8d9-4ed9-86df-56979b6ac88a\") " pod="openshift-ingress-canary/ingress-canary-92vwp" Nov 02 19:52:44.520405 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.520362 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:44.520434 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.520399 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-trusted-ca-bundle\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.520434 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.520429 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmns\" (UniqueName: \"kubernetes.io/projected/fafdd19a-1c68-4429-9519-7da446447921-kube-api-access-6bmns\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.520492 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.520466 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw92p\" (UniqueName: \"kubernetes.io/projected/002e660f-77e7-4384-a7ca-514cc40a4cb6-kube-api-access-mw92p\") pod \"monitoring-plugin-77f4f76594-6qrl5\" (UID: \"002e660f-77e7-4384-a7ca-514cc40a4cb6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:44.520721 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.520706 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-service-ca\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.520781 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.520768 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0170aa95-6bc5-4107-98f6-7beea4227f4f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:44.521174 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.521159 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbxrc\" (UniqueName: \"kubernetes.io/projected/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-api-access-xbxrc\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:44.521540 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.521524 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/114df931-a2e1-4569-95c9-b7cc0c440079-metrics-server-audit-profiles\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.521572 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.521537 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-oauth-serving-cert\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.521676 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.521663 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:44.521749 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.521731 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87tb\" (UniqueName: \"kubernetes.io/projected/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-kube-api-access-c87tb\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.521812 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.521799 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:44.522206 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.522191 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7m7\" (UniqueName: \"kubernetes.io/projected/c10c2efe-3e34-4ca7-b972-680cde439609-kube-api-access-4v7m7\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:52:44.522406 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.522389 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-service-ca\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.522700 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.522681 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.522911 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.522890 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4j5s\" (UniqueName: \"kubernetes.io/projected/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-kube-api-access-s4j5s\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.523176 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.523160 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2f4h\" (UniqueName: \"kubernetes.io/projected/114df931-a2e1-4569-95c9-b7cc0c440079-kube-api-access-j2f4h\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.523295 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.523281 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbrc\" (UniqueName: \"kubernetes.io/projected/f0ed3429-8e54-4f88-b820-751eacadd01a-kube-api-access-cxbrc\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.523667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.523646 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vknpr\" (UniqueName: \"kubernetes.io/projected/367d4910-a962-43b9-a7e6-a58b47636a24-kube-api-access-vknpr\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.523706 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.523681 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t966\" (UniqueName: \"kubernetes.io/projected/0170aa95-6bc5-4107-98f6-7beea4227f4f-kube-api-access-9t966\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:44.524057 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.524041 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc678\" (UniqueName: \"kubernetes.io/projected/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-kube-api-access-qc678\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.524422 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.524402 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8ws\" (UniqueName: \"kubernetes.io/projected/a409cedc-aedc-4130-aad6-4405f1539799-kube-api-access-9x8ws\") pod \"dns-default-jqtjb\" (UID: \"a409cedc-aedc-4130-aad6-4405f1539799\") " pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:44.524924 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.524902 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmg6s\" (UniqueName: \"kubernetes.io/projected/092fd489-4d42-46d1-a178-e95204a400e8-kube-api-access-jmg6s\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.525023 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.525004 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwzr\" (UniqueName: \"kubernetes.io/projected/b2ff0bc2-a4cf-45f3-b870-2526d024b6c9-kube-api-access-gnwzr\") pod \"network-check-target-m2tvk\" (UID: \"b2ff0bc2-a4cf-45f3-b870-2526d024b6c9\") " pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:44.525823 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.525802 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgkrt\" (UniqueName: \"kubernetes.io/projected/a3823ef4-9aae-417c-bc9a-a04253589bf5-kube-api-access-fgkrt\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.526016 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.526002 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb5kf\" (UniqueName: \"kubernetes.io/projected/98798482-f8b5-401e-9aa2-363cda917fba-kube-api-access-bb5kf\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.526069 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.526051 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8mg9\" (UniqueName: \"kubernetes.io/projected/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-kube-api-access-x8mg9\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:44.526111 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.526051 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4j9w\" (UniqueName: \"kubernetes.io/projected/0d60e019-846f-497e-a1fb-1c972295523b-kube-api-access-s4j9w\") pod \"network-metrics-daemon-r67nn\" (UID: \"0d60e019-846f-497e-a1fb-1c972295523b\") " pod="openshift-multus/network-metrics-daemon-r67nn" Nov 02 19:52:44.549983 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.549960 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 19:52:44.550157 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550126 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 19:52:44.550246 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550166 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 19:52:44.550246 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550177 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 19:52:44.550326 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550248 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 19:52:44.550572 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550554 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d5pq8\"" Nov 02 19:52:44.550650 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550577 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 19:52:44.550650 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550634 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:44.550803 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550768 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 19:52:44.550803 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550772 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 19:52:44.550803 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550789 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-94vpabn2q7b5i\"" Nov 02 19:52:44.550981 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550846 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vg4mb\"" Nov 02 19:52:44.550981 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550932 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 19:52:44.550981 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550951 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xjzch\"" Nov 02 19:52:44.550981 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550965 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 19:52:44.550981 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550972 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-t8rrc\"" Nov 02 19:52:44.550981 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550977 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7qs77\"" Nov 02 19:52:44.551199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550993 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 19:52:44.551199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.550996 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 19:52:44.551199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551008 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 19:52:44.551199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551015 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 19:52:44.551199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551016 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-barv9dd6r2s2h\"" Nov 02 19:52:44.551199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551027 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 19:52:44.551199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551039 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5zj4s\"" Nov 02 19:52:44.551199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551064 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-t224c\"" Nov 02 19:52:44.551199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551132 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dnc86\"" Nov 02 19:52:44.551199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551146 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/002e660f-77e7-4384-a7ca-514cc40a4cb6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6qrl5\" (UID: \"002e660f-77e7-4384-a7ca-514cc40a4cb6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:44.551199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551166 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 19:52:44.551199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551192 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jbw4z\"" Nov 02 19:52:44.551521 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551255 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 19:52:44.551521 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551314 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 19:52:44.551521 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.551345 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.552046 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.552024 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.552046 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.552043 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0170aa95-6bc5-4107-98f6-7beea4227f4f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:44.552161 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.552081 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.552221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.552209 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ab821fc-d8d9-4ed9-86df-56979b6ac88a-cert\") pod \"ingress-canary-92vwp\" (UID: \"2ab821fc-d8d9-4ed9-86df-56979b6ac88a\") " pod="openshift-ingress-canary/ingress-canary-92vwp" Nov 02 19:52:44.558407 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.558373 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.558455 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.558394 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0170aa95-6bc5-4107-98f6-7beea4227f4f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-46wd5\" (UID: \"0170aa95-6bc5-4107-98f6-7beea4227f4f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-46wd5" Nov 02 19:52:44.558455 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.558429 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:44.558522 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.558436 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-client-ca-bundle\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.558565 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.558555 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-tls\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.558700 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.558687 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-secret-metrics-server-tls\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.558770 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.558756 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-tls\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.558908 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.558896 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/114df931-a2e1-4569-95c9-b7cc0c440079-secret-metrics-server-client-certs\") pod \"metrics-server-796b7c8599-d2vtr\" (UID: \"114df931-a2e1-4569-95c9-b7cc0c440079\") " pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:44.559438 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.559426 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.559508 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.559493 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-serving-cert\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.560034 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.560017 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c5d86ef2-bb3e-4128-a81e-875fd9f6c527-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-78bc9\" (UID: \"c5d86ef2-bb3e-4128-a81e-875fd9f6c527\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-78bc9" Nov 02 19:52:44.560034 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.560028 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/367d4910-a962-43b9-a7e6-a58b47636a24-secret-grpc-tls\") pod \"thanos-querier-7868584944-ktpmj\" (UID: \"367d4910-a962-43b9-a7e6-a58b47636a24\") " pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:44.561102 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.561082 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-oauth-config\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.561144 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.561099 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-oauth-config\") pod \"console-67c8cb984d-5gwb9\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:44.561190 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.561177 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-serving-cert\") pod \"console-8478dfbff8-ldrtw\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:52:44.572675 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572654 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-socket-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.572740 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572680 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e588ae83-565a-4711-a7b7-d77b8ae24bc8-hosts-file\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:52:44.572740 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572697 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6dde0e70-73a7-4198-831e-875401c4c3cd-dbus\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:44.572740 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572711 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.572740 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572726 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-systemd\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.572926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572745 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-kubelet\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.572926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572769 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysctl-d\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.572926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572790 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6dde0e70-73a7-4198-831e-875401c4c3cd-dbus\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:44.572926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572804 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.572926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572810 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e588ae83-565a-4711-a7b7-d77b8ae24bc8-hosts-file\") pod \"node-resolver-9rglx\" (UID: \"e588ae83-565a-4711-a7b7-d77b8ae24bc8\") " pod="openshift-dns/node-resolver-9rglx" Nov 02 19:52:44.572926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572794 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-socket-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.572926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572816 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-kubelet\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.572926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572834 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-systemd\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.572926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572794 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-lib-modules\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.572926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572908 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-host\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:52:44.572926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572926 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6dde0e70-73a7-4198-831e-875401c4c3cd-kubelet-config\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572940 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysctl-d\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572940 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-lib-modules\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572944 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-wtmp\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572973 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c6f9fda-5459-4d5a-8405-7eab0de2cb1e-host\") pod \"node-ca-dsct6\" (UID: \"2c6f9fda-5459-4d5a-8405-7eab0de2cb1e\") " pod="openshift-image-registry/node-ca-dsct6" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572975 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6dde0e70-73a7-4198-831e-875401c4c3cd-kubelet-config\") pod \"global-pull-secret-syncer-4bpmh\" (UID: \"6dde0e70-73a7-4198-831e-875401c4c3cd\") " pod="kube-system/global-pull-secret-syncer-4bpmh" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.572987 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-etc-kubernetes\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573007 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-etc-kubernetes\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573011 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-log-socket\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573021 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-node-exporter-wtmp\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573031 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-log-socket\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573036 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-cni-netd\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573057 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-kubernetes\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573073 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-socket-dir-parent\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573093 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-hostroot\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573116 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-device-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573134 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-cni-netd\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573140 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-registration-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573162 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-hostroot\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573167 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-system-cni-dir\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573178 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-socket-dir-parent\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573196 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-registration-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573207 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-device-dir\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573204 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-netns\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573219 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-system-cni-dir\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.573216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573226 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-kubernetes\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573240 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysconfig\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573244 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-netns\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573262 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-host\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573279 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-etc-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573279 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysconfig\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573299 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-crio-socket\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573342 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-etc-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573343 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-host\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573354 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef721fd3-36cc-4fc3-b17c-6ae016daf10c-crio-socket\") pod \"insights-runtime-extractor-4b5v5\" (UID: \"ef721fd3-36cc-4fc3-b17c-6ae016daf10c\") " pod="openshift-insights/insights-runtime-extractor-4b5v5" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573361 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573388 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-k8s-cni-cncf-io\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573403 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-var-lib-kubelet\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573419 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-ovn\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573437 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-k8s-cni-cncf-io\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573436 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysctl-conf\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573451 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-var-lib-kubelet\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573468 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573476 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-system-cni-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573485 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-ovn\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573505 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-kubelet\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573511 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-system-cni-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573524 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-conf-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573540 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-var-lib-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573552 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-kubelet\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573557 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573554 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-sysctl-conf\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573583 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-conf-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573585 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-cni-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573589 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-var-lib-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573609 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-openvswitch\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573612 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c10c2efe-3e34-4ca7-b972-680cde439609-host-slash\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573628 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-sys-fs\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573642 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-cnibin\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573659 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c10c2efe-3e34-4ca7-b972-680cde439609-host-slash\") pod \"iptables-alerter-nqhf8\" (UID: \"c10c2efe-3e34-4ca7-b972-680cde439609\") " pod="openshift-network-operator/iptables-alerter-nqhf8" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573661 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-sys\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573701 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-cni-bin\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573707 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-multus-cni-dir\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573708 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-sys-fs\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573708 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-sys\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573707 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-cnibin\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573766 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-cni-bin\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573784 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-cni-multus\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573799 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-multus-certs\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573816 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-systemd-units\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573829 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-systemd\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573842 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-node-log\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573869 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-node-log\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573888 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-run-systemd\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573886 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-var-lib-cni-multus\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573912 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57-host-run-multus-certs\") pod \"multus-z2vn7\" (UID: \"7b5d5ddb-65f0-4e9d-b1f3-7115a1087e57\") " pod="openshift-multus/multus-z2vn7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573909 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573934 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-systemd-units\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573947 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-cni-bin\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573974 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573974 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-run-netns\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574009 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574011 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-run-netns\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.573987 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-cni-bin\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574027 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-slash\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.574072 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574075 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-modprobe-d\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574053 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98798482-f8b5-401e-9aa2-363cda917fba-host-slash\") pod \"ovnkube-node-r6lg7\" (UID: \"98798482-f8b5-401e-9aa2-363cda917fba\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574181 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-etc-modprobe-d\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574076 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d10bb7fa-2eb9-4215-b467-c2165c4c76f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-8s4xk\" (UID: \"d10bb7fa-2eb9-4215-b467-c2165c4c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8s4xk" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574491 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-cnibin\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574507 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-sys\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574521 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-root\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574534 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-run\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574569 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574592 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-sys\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574595 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-cnibin\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574608 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/092fd489-4d42-46d1-a178-e95204a400e8-root\") pod \"node-exporter-287j2\" (UID: \"092fd489-4d42-46d1-a178-e95204a400e8\") " pod="openshift-monitoring/node-exporter-287j2" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574656 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3823ef4-9aae-417c-bc9a-a04253589bf5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7svjs\" (UID: \"a3823ef4-9aae-417c-bc9a-a04253589bf5\") " pod="openshift-multus/multus-additional-cni-plugins-7svjs" Nov 02 19:52:44.575728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.574684 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0ed3429-8e54-4f88-b820-751eacadd01a-run\") pod \"tuned-7m9qt\" (UID: \"f0ed3429-8e54-4f88-b820-751eacadd01a\") " pod="openshift-cluster-node-tuning-operator/tuned-7m9qt" Nov 02 19:52:44.592900 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.592858 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:44.740093 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.740067 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qrl5" Nov 02 19:52:44.769186 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:44.769098 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:45.945431 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:45.945397 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:52:47.366743 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:47.366710 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7868584944-ktpmj" Nov 02 19:52:47.393043 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:47.393020 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jqtjb" Nov 02 19:52:48.518824 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:48.518798 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:49.368941 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:49.368914 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:49.385011 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:49.384988 5561 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:52:50.987516 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:50.987481 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r9jxd" Nov 02 19:52:51.284843 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:51.284768 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-796b7c8599-d2vtr" Nov 02 19:52:52.420602 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:52.420570 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:52:54.287721 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:54.287622 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:54.455580 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:54.455545 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:57.505167 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:57.505132 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:57.752809 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:57.752783 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:57.772717 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:57.772649 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6lg7" Nov 02 19:52:59.504400 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:59.504367 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:52:59.686100 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:52:59.686060 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:03.778962 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:53:03.778922 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:53:08.228708 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.228672 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66cf47b6f4-5drsz"] Nov 02 19:53:08.235106 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.235083 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.286222 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.286186 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66cf47b6f4-5drsz"] Nov 02 19:53:08.351926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.351863 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/535a2425-a2e8-4b72-86d8-95d64f619d7b-tmp\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.351926 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.351932 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-trusted-ca-bundle\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.352135 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.352025 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-config\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.352135 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.352048 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-service-ca\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.352135 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.352084 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68lfd\" (UniqueName: \"kubernetes.io/projected/535a2425-a2e8-4b72-86d8-95d64f619d7b-kube-api-access-68lfd\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.352135 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.352099 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-oauth-serving-cert\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.352255 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.352224 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-serving-cert\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.352255 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.352248 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-oauth-config\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.453342 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.453305 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-oauth-config\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.453342 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.453342 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/535a2425-a2e8-4b72-86d8-95d64f619d7b-tmp\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.453584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.453361 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-trusted-ca-bundle\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.453584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.453451 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-config\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.453584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.453466 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-service-ca\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.453584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.453504 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68lfd\" (UniqueName: \"kubernetes.io/projected/535a2425-a2e8-4b72-86d8-95d64f619d7b-kube-api-access-68lfd\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.453584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.453520 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-oauth-serving-cert\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.453898 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.453763 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/535a2425-a2e8-4b72-86d8-95d64f619d7b-tmp\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.453898 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.453806 5561 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 19:53:08.453898 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.453820 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-serving-cert\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.454379 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.454350 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-config\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.454379 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.454369 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-oauth-serving-cert\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.454559 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.454369 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-service-ca\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.454620 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.454609 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-trusted-ca-bundle\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.456071 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.456050 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-oauth-config\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.456174 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.456134 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-serving-cert\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.507240 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.507178 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68lfd\" (UniqueName: \"kubernetes.io/projected/535a2425-a2e8-4b72-86d8-95d64f619d7b-kube-api-access-68lfd\") pod \"console-66cf47b6f4-5drsz\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.545039 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.545013 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:08.546670 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.546636 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:08.712664 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.712625 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:08.911343 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.911311 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66cf47b6f4-5drsz"] Nov 02 19:53:08.920122 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:53:08.920088 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535a2425_a2e8_4b72_86d8_95d64f619d7b.slice/crio-d049d94143366f3561d80e074ece3dad55068e1ccc4b0e8d2db794bc0e293ac0 WatchSource:0}: Error finding container d049d94143366f3561d80e074ece3dad55068e1ccc4b0e8d2db794bc0e293ac0: Status 404 returned error can't find the container with id d049d94143366f3561d80e074ece3dad55068e1ccc4b0e8d2db794bc0e293ac0 Nov 02 19:53:08.938786 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:08.938767 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:09.111248 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.111216 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:09.387306 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.387195 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8478dfbff8-ldrtw" podUID="fafdd19a-1c68-4429-9519-7da446447921" containerName="console" containerID="cri-o://519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7" gracePeriod=15 Nov 02 19:53:09.460592 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.460559 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cf47b6f4-5drsz" event={"ID":"535a2425-a2e8-4b72-86d8-95d64f619d7b","Type":"ContainerStarted","Data":"044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024"} Nov 02 19:53:09.460745 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.460626 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cf47b6f4-5drsz" event={"ID":"535a2425-a2e8-4b72-86d8-95d64f619d7b","Type":"ContainerStarted","Data":"d049d94143366f3561d80e074ece3dad55068e1ccc4b0e8d2db794bc0e293ac0"} Nov 02 19:53:09.570577 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.570526 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66cf47b6f4-5drsz" podStartSLOduration=1.5705136849999999 podStartE2EDuration="1.570513685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:09.570177784 +0000 UTC m=+26.377962001" watchObservedRunningTime="2026-11-02 19:53:09.570513685 +0000 UTC m=+26.378297893" Nov 02 19:53:09.625942 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.625913 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:53:09.763845 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.763810 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-oauth-serving-cert\") pod \"fafdd19a-1c68-4429-9519-7da446447921\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " Nov 02 19:53:09.764033 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.763869 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-serving-cert\") pod \"fafdd19a-1c68-4429-9519-7da446447921\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " Nov 02 19:53:09.764033 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.763926 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-console-config\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-console-config\") pod \"fafdd19a-1c68-4429-9519-7da446447921\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " Nov 02 19:53:09.764033 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.763950 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fafdd19a-1c68-4429-9519-7da446447921-kube-api-access-6bmns\" (UniqueName: \"kubernetes.io/projected/fafdd19a-1c68-4429-9519-7da446447921-kube-api-access-6bmns\") pod \"fafdd19a-1c68-4429-9519-7da446447921\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " Nov 02 19:53:09.764033 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.763988 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fafdd19a-1c68-4429-9519-7da446447921-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafdd19a-1c68-4429-9519-7da446447921-tmp\") pod \"fafdd19a-1c68-4429-9519-7da446447921\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " Nov 02 19:53:09.764033 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.764018 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-service-ca\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-service-ca\") pod \"fafdd19a-1c68-4429-9519-7da446447921\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " Nov 02 19:53:09.764033 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.764035 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-oauth-config\") pod \"fafdd19a-1c68-4429-9519-7da446447921\" (UID: \"fafdd19a-1c68-4429-9519-7da446447921\") " Nov 02 19:53:09.764394 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.764358 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fafdd19a-1c68-4429-9519-7da446447921-tmp" pod "fafdd19a-1c68-4429-9519-7da446447921" (UID: "fafdd19a-1c68-4429-9519-7da446447921"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:09.764394 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.764375 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-console-config" pod "fafdd19a-1c68-4429-9519-7da446447921" (UID: "fafdd19a-1c68-4429-9519-7da446447921"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:09.764499 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.764376 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-oauth-serving-cert" pod "fafdd19a-1c68-4429-9519-7da446447921" (UID: "fafdd19a-1c68-4429-9519-7da446447921"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:09.764499 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.764469 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-service-ca" pod "fafdd19a-1c68-4429-9519-7da446447921" (UID: "fafdd19a-1c68-4429-9519-7da446447921"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:09.766337 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.766316 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-oauth-config" pod "fafdd19a-1c68-4429-9519-7da446447921" (UID: "fafdd19a-1c68-4429-9519-7da446447921"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:09.766436 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.766420 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-serving-cert" pod "fafdd19a-1c68-4429-9519-7da446447921" (UID: "fafdd19a-1c68-4429-9519-7da446447921"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:09.766472 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.766434 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafdd19a-1c68-4429-9519-7da446447921-kube-api-access-6bmns" pod "fafdd19a-1c68-4429-9519-7da446447921" (UID: "fafdd19a-1c68-4429-9519-7da446447921"). InnerVolumeSpecName "kube-api-access-6bmns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:09.864806 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.864763 5561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-oauth-serving-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:09.864806 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.864803 5561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-serving-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:09.865043 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.864818 5561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-console-config\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:09.865043 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.864830 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bmns\" (UniqueName: \"kubernetes.io/projected/fafdd19a-1c68-4429-9519-7da446447921-kube-api-access-6bmns\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:09.865043 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.864845 5561 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fafdd19a-1c68-4429-9519-7da446447921-tmp\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:09.865043 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.864859 5561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fafdd19a-1c68-4429-9519-7da446447921-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:09.865043 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:09.864870 5561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fafdd19a-1c68-4429-9519-7da446447921-console-oauth-config\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:10.464887 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:10.464845 5561 generic.go:356] "Generic (PLEG): container finished" podID="fafdd19a-1c68-4429-9519-7da446447921" containerID="519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7" exitCode=0 Nov 02 19:53:10.465323 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:10.464909 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8478dfbff8-ldrtw" Nov 02 19:53:10.465323 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:10.464939 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8478dfbff8-ldrtw" event={"ID":"fafdd19a-1c68-4429-9519-7da446447921","Type":"ContainerDied","Data":"519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7"} Nov 02 19:53:10.465323 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:10.464989 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8478dfbff8-ldrtw" event={"ID":"fafdd19a-1c68-4429-9519-7da446447921","Type":"ContainerDied","Data":"263cb0b0be40e4059ac1a4f5018e966d5ee3bfd6a417af4702b2846d7ed335d8"} Nov 02 19:53:10.465323 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:10.465011 5561 scope.go:122] "RemoveContainer" containerID="519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7" Nov 02 19:53:10.473405 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:10.473383 5561 scope.go:122] "RemoveContainer" containerID="519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7" Nov 02 19:53:10.473659 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:53:10.473641 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7\": container with ID starting with 519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7 not found: ID does not exist" containerID="519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7" Nov 02 19:53:10.473725 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:10.473668 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7"} err="failed to get container status \"519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7\": rpc error: code = NotFound desc = could not find container \"519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7\": container with ID starting with 519ca774c11e2ad5203a8ac1c5ff3b7e605ca98415d7f44db73b1a5eee863da7 not found: ID does not exist" Nov 02 19:53:10.618053 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:10.618021 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8478dfbff8-ldrtw"] Nov 02 19:53:10.665483 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:10.665450 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8478dfbff8-ldrtw"] Nov 02 19:53:11.311102 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:11.311067 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fafdd19a-1c68-4429-9519-7da446447921" path="/var/lib/kubelet/pods/fafdd19a-1c68-4429-9519-7da446447921/volumes" Nov 02 19:53:14.486624 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:14.486585 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:14.656454 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:14.656417 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:15.948640 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:15.948612 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m2tvk" Nov 02 19:53:18.545588 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:18.545552 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:18.545969 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:18.545633 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:18.550386 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:18.550367 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:19.499423 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:19.499396 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:53:19.657228 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:19.657196 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67c8cb984d-5gwb9"] Nov 02 19:53:33.777819 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:53:33.777778 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:53:41.132164 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:41.132133 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:41.363844 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:41.363816 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:44.677279 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:44.677213 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-67c8cb984d-5gwb9" podUID="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" containerName="console" containerID="cri-o://75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7" gracePeriod=15 Nov 02 19:53:44.915040 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:44.915017 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:53:45.044856 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.044780 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-tmp\") pod \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " Nov 02 19:53:45.044856 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.044826 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-trusted-ca-bundle\") pod \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " Nov 02 19:53:45.044856 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.044852 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-serving-cert\") pod \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " Nov 02 19:53:45.045080 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.044905 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-service-ca\") pod \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " Nov 02 19:53:45.045166 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.045104 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-tmp" pod "f2ebae9e-27da-4d11-93c6-acc7657cd4d5" (UID: "f2ebae9e-27da-4d11-93c6-acc7657cd4d5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:45.045166 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.045122 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-kube-api-access-qc678\" (UniqueName: \"kubernetes.io/projected/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-kube-api-access-qc678\") pod \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " Nov 02 19:53:45.045251 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.045183 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-config\") pod \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " Nov 02 19:53:45.045251 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.045207 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-oauth-config\") pod \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " Nov 02 19:53:45.045251 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.045225 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-oauth-serving-cert\") pod \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\" (UID: \"f2ebae9e-27da-4d11-93c6-acc7657cd4d5\") " Nov 02 19:53:45.045411 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.045245 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-trusted-ca-bundle" pod "f2ebae9e-27da-4d11-93c6-acc7657cd4d5" (UID: "f2ebae9e-27da-4d11-93c6-acc7657cd4d5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:45.045411 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.045300 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-service-ca" pod "f2ebae9e-27da-4d11-93c6-acc7657cd4d5" (UID: "f2ebae9e-27da-4d11-93c6-acc7657cd4d5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:45.045594 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.045575 5561 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-tmp\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:45.045648 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.045585 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-config" pod "f2ebae9e-27da-4d11-93c6-acc7657cd4d5" (UID: "f2ebae9e-27da-4d11-93c6-acc7657cd4d5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:45.045648 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.045600 5561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-trusted-ca-bundle\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:45.045648 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.045617 5561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:45.045648 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.045628 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-oauth-serving-cert" pod "f2ebae9e-27da-4d11-93c6-acc7657cd4d5" (UID: "f2ebae9e-27da-4d11-93c6-acc7657cd4d5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:45.047316 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.047290 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-oauth-config" pod "f2ebae9e-27da-4d11-93c6-acc7657cd4d5" (UID: "f2ebae9e-27da-4d11-93c6-acc7657cd4d5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:45.047316 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.047301 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-kube-api-access-qc678" pod "f2ebae9e-27da-4d11-93c6-acc7657cd4d5" (UID: "f2ebae9e-27da-4d11-93c6-acc7657cd4d5"). InnerVolumeSpecName "kube-api-access-qc678". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:45.047444 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.047359 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-serving-cert" pod "f2ebae9e-27da-4d11-93c6-acc7657cd4d5" (UID: "f2ebae9e-27da-4d11-93c6-acc7657cd4d5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:45.147057 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.147019 5561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-serving-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:45.147057 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.147051 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc678\" (UniqueName: \"kubernetes.io/projected/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-kube-api-access-qc678\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:45.147057 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.147063 5561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-config\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:45.147294 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.147072 5561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-console-oauth-config\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:45.147294 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.147082 5561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebae9e-27da-4d11-93c6-acc7657cd4d5-oauth-serving-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:45.430134 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.430099 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:45.585414 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.585335 5561 generic.go:356] "Generic (PLEG): container finished" podID="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" containerID="75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7" exitCode=0 Nov 02 19:53:45.585414 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.585385 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c8cb984d-5gwb9" Nov 02 19:53:45.585576 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.585413 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c8cb984d-5gwb9" event={"ID":"f2ebae9e-27da-4d11-93c6-acc7657cd4d5","Type":"ContainerDied","Data":"75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7"} Nov 02 19:53:45.585576 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.585464 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c8cb984d-5gwb9" event={"ID":"f2ebae9e-27da-4d11-93c6-acc7657cd4d5","Type":"ContainerDied","Data":"1aa740e4cd6c008dcabbca2e4cdf195848a227921fde726f1ba39f42d5381d92"} Nov 02 19:53:45.585576 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.585485 5561 scope.go:122] "RemoveContainer" containerID="75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7" Nov 02 19:53:45.593108 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.593092 5561 scope.go:122] "RemoveContainer" containerID="75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7" Nov 02 19:53:45.593373 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:53:45.593355 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7\": container with ID starting with 75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7 not found: ID does not exist" containerID="75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7" Nov 02 19:53:45.593435 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.593387 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7"} err="failed to get container status \"75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7\": rpc error: code = NotFound desc = could not find container \"75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7\": container with ID starting with 75d2778f9a1ddc856565f55c9a09dfcc1fc109a2781b660bd5490fc2138225f7 not found: ID does not exist" Nov 02 19:53:45.740232 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.740199 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67c8cb984d-5gwb9"] Nov 02 19:53:45.787640 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:45.787609 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67c8cb984d-5gwb9"] Nov 02 19:53:47.310311 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:47.310276 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" path="/var/lib/kubelet/pods/f2ebae9e-27da-4d11-93c6-acc7657cd4d5/volumes" Nov 02 19:53:56.059506 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.059467 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-75648b47dc-qcgwq"] Nov 02 19:53:56.060127 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.059939 5561 cpu_manager.go:414] "containerMap: removing container" podUID="fafdd19a-1c68-4429-9519-7da446447921" containerName="console" Nov 02 19:53:56.060127 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.059959 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fafdd19a-1c68-4429-9519-7da446447921" containerName="console" Nov 02 19:53:56.060127 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.059979 5561 cpu_manager.go:414] "containerMap: removing container" podUID="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" containerName="console" Nov 02 19:53:56.060127 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.059986 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" containerName="console" Nov 02 19:53:56.060127 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.060056 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="fafdd19a-1c68-4429-9519-7da446447921" containerName="console" Nov 02 19:53:56.060127 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.060067 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ebae9e-27da-4d11-93c6-acc7657cd4d5" containerName="console" Nov 02 19:53:56.062822 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.062800 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.128957 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.128925 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd3397d-907d-4c3c-9a2b-e9a46702279b-oauth-serving-cert\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.129121 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.128967 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd3397d-907d-4c3c-9a2b-e9a46702279b-console-oauth-config\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.129121 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.128984 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd3397d-907d-4c3c-9a2b-e9a46702279b-service-ca\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.129121 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.129041 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd3397d-907d-4c3c-9a2b-e9a46702279b-trusted-ca-bundle\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.129121 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.129085 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd3397d-907d-4c3c-9a2b-e9a46702279b-console-serving-cert\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.129121 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.129116 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bd3397d-907d-4c3c-9a2b-e9a46702279b-tmp\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.129337 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.129167 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd3397d-907d-4c3c-9a2b-e9a46702279b-console-config\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.129337 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.129235 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w6lq\" (UniqueName: \"kubernetes.io/projected/7bd3397d-907d-4c3c-9a2b-e9a46702279b-kube-api-access-9w6lq\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.155132 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.155104 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75648b47dc-qcgwq"] Nov 02 19:53:56.230007 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.229970 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd3397d-907d-4c3c-9a2b-e9a46702279b-console-config\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.230180 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.230017 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w6lq\" (UniqueName: \"kubernetes.io/projected/7bd3397d-907d-4c3c-9a2b-e9a46702279b-kube-api-access-9w6lq\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.230180 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.230048 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd3397d-907d-4c3c-9a2b-e9a46702279b-oauth-serving-cert\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.230180 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.230087 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd3397d-907d-4c3c-9a2b-e9a46702279b-console-oauth-config\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.230180 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.230113 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd3397d-907d-4c3c-9a2b-e9a46702279b-service-ca\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.230180 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.230135 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd3397d-907d-4c3c-9a2b-e9a46702279b-trusted-ca-bundle\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.230180 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.230167 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd3397d-907d-4c3c-9a2b-e9a46702279b-console-serving-cert\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.230492 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.230296 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bd3397d-907d-4c3c-9a2b-e9a46702279b-tmp\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.230754 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.230730 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bd3397d-907d-4c3c-9a2b-e9a46702279b-tmp\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.230865 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.230842 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bd3397d-907d-4c3c-9a2b-e9a46702279b-console-config\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.230865 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.230848 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bd3397d-907d-4c3c-9a2b-e9a46702279b-oauth-serving-cert\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.231064 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.231041 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd3397d-907d-4c3c-9a2b-e9a46702279b-service-ca\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.231192 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.231174 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bd3397d-907d-4c3c-9a2b-e9a46702279b-trusted-ca-bundle\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.232518 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.232493 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bd3397d-907d-4c3c-9a2b-e9a46702279b-console-oauth-config\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.232614 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.232597 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd3397d-907d-4c3c-9a2b-e9a46702279b-console-serving-cert\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.284226 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.284193 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w6lq\" (UniqueName: \"kubernetes.io/projected/7bd3397d-907d-4c3c-9a2b-e9a46702279b-kube-api-access-9w6lq\") pod \"console-75648b47dc-qcgwq\" (UID: \"7bd3397d-907d-4c3c-9a2b-e9a46702279b\") " pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.374106 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.374023 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:53:56.385666 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.385635 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:56.563951 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.563916 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:56.739016 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.738981 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75648b47dc-qcgwq"] Nov 02 19:53:56.748474 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:53:56.748444 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd3397d_907d_4c3c_9a2b_e9a46702279b.slice/crio-1e524561d666f7ff1dc3aa5066d3aa89ca053deff586b3e2733d6008678beacb WatchSource:0}: Error finding container 1e524561d666f7ff1dc3aa5066d3aa89ca053deff586b3e2733d6008678beacb: Status 404 returned error can't find the container with id 1e524561d666f7ff1dc3aa5066d3aa89ca053deff586b3e2733d6008678beacb Nov 02 19:53:56.792973 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:56.792947 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:57.002439 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:57.002353 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:57.626553 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:57.626518 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75648b47dc-qcgwq" event={"ID":"7bd3397d-907d-4c3c-9a2b-e9a46702279b","Type":"ContainerStarted","Data":"12a94c1dab34ca06f6fb749f02bd985ef978f43235dd69ddeb86cf9e0e656bfb"} Nov 02 19:53:57.626955 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:57.626572 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75648b47dc-qcgwq" event={"ID":"7bd3397d-907d-4c3c-9a2b-e9a46702279b","Type":"ContainerStarted","Data":"1e524561d666f7ff1dc3aa5066d3aa89ca053deff586b3e2733d6008678beacb"} Nov 02 19:53:57.736387 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:57.736335 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-75648b47dc-qcgwq" podStartSLOduration=1.7363201529999999 podStartE2EDuration="1.736320153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:57.735913149 +0000 UTC m=+74.543697390" watchObservedRunningTime="2026-11-02 19:53:57.736320153 +0000 UTC m=+74.544104363" Nov 02 19:53:57.965665 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:57.965637 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:53:58.133248 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:53:58.133216 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:03.778351 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:54:03.778311 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:54:06.374314 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:06.374274 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:54:06.374709 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:06.374331 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:54:06.379066 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:06.379044 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:54:06.659836 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:06.659810 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75648b47dc-qcgwq" Nov 02 19:54:06.797860 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:06.797832 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66cf47b6f4-5drsz"] Nov 02 19:54:14.518728 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:14.518691 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:14.719398 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:14.719362 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:54:31.816758 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:31.816674 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-66cf47b6f4-5drsz" podUID="535a2425-a2e8-4b72-86d8-95d64f619d7b" containerName="console" containerID="cri-o://044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024" gracePeriod=15 Nov 02 19:54:32.070434 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.070376 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:54:32.212668 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.212631 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-oauth-config\") pod \"535a2425-a2e8-4b72-86d8-95d64f619d7b\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " Nov 02 19:54:32.212837 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.212679 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-service-ca\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-service-ca\") pod \"535a2425-a2e8-4b72-86d8-95d64f619d7b\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " Nov 02 19:54:32.212837 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.212718 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535a2425-a2e8-4b72-86d8-95d64f619d7b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/535a2425-a2e8-4b72-86d8-95d64f619d7b-tmp\") pod \"535a2425-a2e8-4b72-86d8-95d64f619d7b\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " Nov 02 19:54:32.212837 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.212771 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-serving-cert\") pod \"535a2425-a2e8-4b72-86d8-95d64f619d7b\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " Nov 02 19:54:32.212837 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.212820 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-oauth-serving-cert\") pod \"535a2425-a2e8-4b72-86d8-95d64f619d7b\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " Nov 02 19:54:32.213069 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.212861 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-config\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-config\") pod \"535a2425-a2e8-4b72-86d8-95d64f619d7b\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " Nov 02 19:54:32.213069 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.212926 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535a2425-a2e8-4b72-86d8-95d64f619d7b-kube-api-access-68lfd\" (UniqueName: \"kubernetes.io/projected/535a2425-a2e8-4b72-86d8-95d64f619d7b-kube-api-access-68lfd\") pod \"535a2425-a2e8-4b72-86d8-95d64f619d7b\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " Nov 02 19:54:32.213069 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.212965 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-trusted-ca-bundle\") pod \"535a2425-a2e8-4b72-86d8-95d64f619d7b\" (UID: \"535a2425-a2e8-4b72-86d8-95d64f619d7b\") " Nov 02 19:54:32.213069 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.213045 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535a2425-a2e8-4b72-86d8-95d64f619d7b-tmp" pod "535a2425-a2e8-4b72-86d8-95d64f619d7b" (UID: "535a2425-a2e8-4b72-86d8-95d64f619d7b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:32.213269 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.213180 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-service-ca" pod "535a2425-a2e8-4b72-86d8-95d64f619d7b" (UID: "535a2425-a2e8-4b72-86d8-95d64f619d7b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:32.213269 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.213240 5561 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/535a2425-a2e8-4b72-86d8-95d64f619d7b-tmp\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:32.213442 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.213411 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-config" pod "535a2425-a2e8-4b72-86d8-95d64f619d7b" (UID: "535a2425-a2e8-4b72-86d8-95d64f619d7b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:32.213526 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.213460 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-trusted-ca-bundle" pod "535a2425-a2e8-4b72-86d8-95d64f619d7b" (UID: "535a2425-a2e8-4b72-86d8-95d64f619d7b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:32.213526 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.213470 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-oauth-serving-cert" pod "535a2425-a2e8-4b72-86d8-95d64f619d7b" (UID: "535a2425-a2e8-4b72-86d8-95d64f619d7b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:54:32.214939 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.214917 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-oauth-config" pod "535a2425-a2e8-4b72-86d8-95d64f619d7b" (UID: "535a2425-a2e8-4b72-86d8-95d64f619d7b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:54:32.215033 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.214965 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535a2425-a2e8-4b72-86d8-95d64f619d7b-kube-api-access-68lfd" pod "535a2425-a2e8-4b72-86d8-95d64f619d7b" (UID: "535a2425-a2e8-4b72-86d8-95d64f619d7b"). InnerVolumeSpecName "kube-api-access-68lfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:32.215033 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.215002 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-serving-cert" pod "535a2425-a2e8-4b72-86d8-95d64f619d7b" (UID: "535a2425-a2e8-4b72-86d8-95d64f619d7b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:54:32.313992 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.313951 5561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-serving-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:32.313992 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.313987 5561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-oauth-serving-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:32.313992 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.313998 5561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-config\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:32.314238 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.314007 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68lfd\" (UniqueName: \"kubernetes.io/projected/535a2425-a2e8-4b72-86d8-95d64f619d7b-kube-api-access-68lfd\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:32.314238 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.314018 5561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-trusted-ca-bundle\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:32.314238 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.314026 5561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/535a2425-a2e8-4b72-86d8-95d64f619d7b-console-oauth-config\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:32.314238 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.314035 5561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/535a2425-a2e8-4b72-86d8-95d64f619d7b-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:32.743010 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.742975 5561 generic.go:356] "Generic (PLEG): container finished" podID="535a2425-a2e8-4b72-86d8-95d64f619d7b" containerID="044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024" exitCode=0 Nov 02 19:54:32.743188 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.743021 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cf47b6f4-5drsz" Nov 02 19:54:32.743188 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.743047 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cf47b6f4-5drsz" event={"ID":"535a2425-a2e8-4b72-86d8-95d64f619d7b","Type":"ContainerDied","Data":"044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024"} Nov 02 19:54:32.743188 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.743091 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cf47b6f4-5drsz" event={"ID":"535a2425-a2e8-4b72-86d8-95d64f619d7b","Type":"ContainerDied","Data":"d049d94143366f3561d80e074ece3dad55068e1ccc4b0e8d2db794bc0e293ac0"} Nov 02 19:54:32.743188 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.743112 5561 scope.go:122] "RemoveContainer" containerID="044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024" Nov 02 19:54:32.751500 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.751483 5561 scope.go:122] "RemoveContainer" containerID="044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024" Nov 02 19:54:32.751743 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:54:32.751725 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024\": container with ID starting with 044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024 not found: ID does not exist" containerID="044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024" Nov 02 19:54:32.751790 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.751758 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024"} err="failed to get container status \"044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024\": rpc error: code = NotFound desc = could not find container \"044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024\": container with ID starting with 044ab3baaf4c5f1812ab8a4038dc47c62d486e7f7d6c0ba3babb84dde29be024 not found: ID does not exist" Nov 02 19:54:32.898266 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.898229 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66cf47b6f4-5drsz"] Nov 02 19:54:32.945956 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:32.945918 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66cf47b6f4-5drsz"] Nov 02 19:54:33.310475 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:33.310443 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="535a2425-a2e8-4b72-86d8-95d64f619d7b" path="/var/lib/kubelet/pods/535a2425-a2e8-4b72-86d8-95d64f619d7b/volumes" Nov 02 19:54:33.777335 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:54:33.777297 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:54:41.464214 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:54:41.464176 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:55:03.781246 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:55:03.781199 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:55:33.778684 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:55:33.778649 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:55:41.800655 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:55:41.800621 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:03.777603 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:56:03.777526 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:56:14.582121 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:14.582084 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:17.483574 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:17.483531 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:17.673462 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:17.673424 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:32.321231 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:32.321196 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:32.553129 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:32.553090 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:33.105242 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.105209 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4"] Nov 02 19:56:33.105521 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.105509 5561 cpu_manager.go:414] "containerMap: removing container" podUID="535a2425-a2e8-4b72-86d8-95d64f619d7b" containerName="console" Nov 02 19:56:33.105564 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.105523 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="535a2425-a2e8-4b72-86d8-95d64f619d7b" containerName="console" Nov 02 19:56:33.105597 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.105581 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="535a2425-a2e8-4b72-86d8-95d64f619d7b" containerName="console" Nov 02 19:56:33.108516 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.108500 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:33.194290 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.194252 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4\" (UID: \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:33.194290 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.194292 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4\" (UID: \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:33.194528 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.194316 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjg8c\" (UniqueName: \"kubernetes.io/projected/fbc884c5-6ca1-40ec-a82f-36815f9173e2-kube-api-access-tjg8c\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4\" (UID: \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:33.206201 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.206168 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 19:56:33.206333 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.206207 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 19:56:33.206333 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.206273 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-kf5ws\"" Nov 02 19:56:33.206937 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.206924 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4"] Nov 02 19:56:33.207253 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.207237 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k"] Nov 02 19:56:33.210467 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.210452 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:56:33.260026 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.259990 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-66fnv\"" Nov 02 19:56:33.276589 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.276564 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k"] Nov 02 19:56:33.295250 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.295225 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4\" (UID: \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:33.295365 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.295262 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjg8c\" (UniqueName: \"kubernetes.io/projected/fbc884c5-6ca1-40ec-a82f-36815f9173e2-kube-api-access-tjg8c\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4\" (UID: \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:33.295365 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.295293 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k\" (UID: \"04bc62e6-903c-4f1e-9688-2e4713c7006d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:56:33.295365 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.295315 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5zfj\" (UniqueName: \"kubernetes.io/projected/04bc62e6-903c-4f1e-9688-2e4713c7006d-kube-api-access-m5zfj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k\" (UID: \"04bc62e6-903c-4f1e-9688-2e4713c7006d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:56:33.295365 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.295346 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k\" (UID: \"04bc62e6-903c-4f1e-9688-2e4713c7006d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:56:33.295365 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.295366 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4\" (UID: \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:33.295580 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.295563 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4\" (UID: \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:33.295633 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.295618 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4\" (UID: \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:33.355493 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.355419 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjg8c\" (UniqueName: \"kubernetes.io/projected/fbc884c5-6ca1-40ec-a82f-36815f9173e2-kube-api-access-tjg8c\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4\" (UID: \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:33.396096 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.396065 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k\" (UID: \"04bc62e6-903c-4f1e-9688-2e4713c7006d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:56:33.396293 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.396104 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5zfj\" (UniqueName: \"kubernetes.io/projected/04bc62e6-903c-4f1e-9688-2e4713c7006d-kube-api-access-m5zfj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k\" (UID: \"04bc62e6-903c-4f1e-9688-2e4713c7006d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:56:33.396293 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.396231 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k\" (UID: \"04bc62e6-903c-4f1e-9688-2e4713c7006d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:56:33.396527 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.396502 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k\" (UID: \"04bc62e6-903c-4f1e-9688-2e4713c7006d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:56:33.396592 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.396524 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k\" (UID: \"04bc62e6-903c-4f1e-9688-2e4713c7006d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:56:33.424212 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.424182 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:33.450870 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.450831 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5zfj\" (UniqueName: \"kubernetes.io/projected/04bc62e6-903c-4f1e-9688-2e4713c7006d-kube-api-access-m5zfj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k\" (UID: \"04bc62e6-903c-4f1e-9688-2e4713c7006d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:56:33.454426 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.454401 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b"] Nov 02 19:56:33.457952 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.457934 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.497522 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.497451 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/639bf58d-c6c3-41ec-a22f-939b92e0b24e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.497522 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.497513 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.497755 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.497545 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.497755 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.497572 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.497755 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.497598 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.497755 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.497696 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.497755 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.497735 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvj6x\" (UniqueName: \"kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-kube-api-access-nvj6x\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.509293 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.509267 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 19:56:33.509412 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.509334 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 19:56:33.512049 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.512032 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 19:56:33.518417 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.518343 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 19:56:33.518417 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.518393 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-42zng\"" Nov 02 19:56:33.518587 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.518517 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 19:56:33.519681 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.519658 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b"] Nov 02 19:56:33.526373 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.526351 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:56:33.540482 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.540450 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:33.598348 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.598313 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.598506 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.598367 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.598506 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.598398 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.598506 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.598425 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.598662 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.598528 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.598662 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.598560 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvj6x\" (UniqueName: \"kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-kube-api-access-nvj6x\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.598662 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.598616 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/639bf58d-c6c3-41ec-a22f-939b92e0b24e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.599136 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.599107 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.600860 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.600836 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.600969 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.600895 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.601167 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.601139 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.601360 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.601345 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/639bf58d-c6c3-41ec-a22f-939b92e0b24e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.609566 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.609524 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 19:56:33.656172 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.656142 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.656353 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.656334 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvj6x\" (UniqueName: \"kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-kube-api-access-nvj6x\") pod \"istiod-openshift-gateway-7876bfcb74-mkg8b\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.748731 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.748677 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:33.772914 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.772856 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:33.781832 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:56:33.781798 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:56:33.797308 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.797281 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4"] Nov 02 19:56:33.811903 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:56:33.810748 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc884c5_6ca1_40ec_a82f_36815f9173e2.slice/crio-3c2863e81fe13ec0925e72b852c29bb5c3ea54b2c24b94d849c15708ebdb442e WatchSource:0}: Error finding container 3c2863e81fe13ec0925e72b852c29bb5c3ea54b2c24b94d849c15708ebdb442e: Status 404 returned error can't find the container with id 3c2863e81fe13ec0925e72b852c29bb5c3ea54b2c24b94d849c15708ebdb442e Nov 02 19:56:33.818870 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.818846 5561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:56:33.907836 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:33.907804 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k"] Nov 02 19:56:33.916065 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:56:33.916036 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bc62e6_903c_4f1e_9688_2e4713c7006d.slice/crio-696d11618e20359a86fb30858ffbfdaf68fd5e495906aca8f18906280e666400 WatchSource:0}: Error finding container 696d11618e20359a86fb30858ffbfdaf68fd5e495906aca8f18906280e666400: Status 404 returned error can't find the container with id 696d11618e20359a86fb30858ffbfdaf68fd5e495906aca8f18906280e666400 Nov 02 19:56:34.148588 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:34.148548 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" event={"ID":"fbc884c5-6ca1-40ec-a82f-36815f9173e2","Type":"ContainerStarted","Data":"3c2863e81fe13ec0925e72b852c29bb5c3ea54b2c24b94d849c15708ebdb442e"} Nov 02 19:56:34.149802 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:34.149763 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" event={"ID":"04bc62e6-903c-4f1e-9688-2e4713c7006d","Type":"ContainerStarted","Data":"696d11618e20359a86fb30858ffbfdaf68fd5e495906aca8f18906280e666400"} Nov 02 19:56:34.151201 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:34.151178 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b"] Nov 02 19:56:34.159368 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:56:34.159317 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639bf58d_c6c3_41ec_a22f_939b92e0b24e.slice/crio-05aad7b22e9bd3f010f6ff4665dc519c06316e33fae5451f6a7edf3333807c09 WatchSource:0}: Error finding container 05aad7b22e9bd3f010f6ff4665dc519c06316e33fae5451f6a7edf3333807c09: Status 404 returned error can't find the container with id 05aad7b22e9bd3f010f6ff4665dc519c06316e33fae5451f6a7edf3333807c09 Nov 02 19:56:34.434173 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:34.434097 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:35.155476 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:35.155430 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" event={"ID":"639bf58d-c6c3-41ec-a22f-939b92e0b24e","Type":"ContainerStarted","Data":"05aad7b22e9bd3f010f6ff4665dc519c06316e33fae5451f6a7edf3333807c09"} Nov 02 19:56:38.896497 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:38.896449 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:38.896814 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:38.896534 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 19:56:39.176860 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:39.176823 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" event={"ID":"639bf58d-c6c3-41ec-a22f-939b92e0b24e","Type":"ContainerStarted","Data":"6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c"} Nov 02 19:56:39.177081 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:39.177061 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:39.179219 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:39.179165 5561 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-mkg8b container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:56:39.179428 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:39.179218 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" podUID="639bf58d-c6c3-41ec-a22f-939b92e0b24e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:56:39.287936 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:39.287864 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" podStartSLOduration=1.553682182 podStartE2EDuration="6.287850019s" totalImagesPullingTime="4.734167837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:39.286841125 +0000 UTC m=+236.094625366" watchObservedRunningTime="2026-11-02 19:56:39.287850019 +0000 UTC m=+236.095634232" Nov 02 19:56:40.181973 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:40.181937 5561 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-mkg8b container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 19:56:40.182474 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:40.182029 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" podUID="639bf58d-c6c3-41ec-a22f-939b92e0b24e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 19:56:40.493010 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:40.492866 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:40.712582 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:40.712537 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:41.177617 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:41.177391 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:41.456205 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:41.456114 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:41.626432 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:41.626397 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:42.117814 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:42.117715 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:42.189447 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:42.189414 5561 generic.go:356] "Generic (PLEG): container finished" podID="fbc884c5-6ca1-40ec-a82f-36815f9173e2" containerID="09de9082b30c1b822438f429c306d6d429f4209293fda30d1aadb08e6ae5be99" exitCode=0 Nov 02 19:56:42.189625 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:42.189474 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" event={"ID":"fbc884c5-6ca1-40ec-a82f-36815f9173e2","Type":"ContainerDied","Data":"09de9082b30c1b822438f429c306d6d429f4209293fda30d1aadb08e6ae5be99"} Nov 02 19:56:42.371449 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:42.371362 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:43.181751 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:43.181711 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:56:43.194566 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:43.194539 5561 generic.go:356] "Generic (PLEG): container finished" podID="04bc62e6-903c-4f1e-9688-2e4713c7006d" containerID="b02149f84b977a6df16071ea54e8cb6ad3a642c1947993787cc88cd723a88be6" exitCode=0 Nov 02 19:56:43.194703 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:43.194630 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" event={"ID":"04bc62e6-903c-4f1e-9688-2e4713c7006d","Type":"ContainerDied","Data":"b02149f84b977a6df16071ea54e8cb6ad3a642c1947993787cc88cd723a88be6"} Nov 02 19:56:45.251605 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:45.251570 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:56:46.208383 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:46.208341 5561 generic.go:356] "Generic (PLEG): container finished" podID="fbc884c5-6ca1-40ec-a82f-36815f9173e2" containerID="10aca84e976618e540f72dff808f903994349762886e50bc55ffd41b259ea08b" exitCode=0 Nov 02 19:56:46.208584 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:46.208408 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" event={"ID":"fbc884c5-6ca1-40ec-a82f-36815f9173e2","Type":"ContainerDied","Data":"10aca84e976618e540f72dff808f903994349762886e50bc55ffd41b259ea08b"} Nov 02 19:56:47.215151 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:47.215109 5561 generic.go:356] "Generic (PLEG): container finished" podID="04bc62e6-903c-4f1e-9688-2e4713c7006d" containerID="6f4c7061c0ea74e7b0ff503a9cfe0c2bfbbddf93079caad3972d6b7fa7b9f444" exitCode=0 Nov 02 19:56:47.215575 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:47.215160 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" event={"ID":"04bc62e6-903c-4f1e-9688-2e4713c7006d","Type":"ContainerDied","Data":"6f4c7061c0ea74e7b0ff503a9cfe0c2bfbbddf93079caad3972d6b7fa7b9f444"} Nov 02 19:56:53.241756 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:53.241715 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" event={"ID":"fbc884c5-6ca1-40ec-a82f-36815f9173e2","Type":"ContainerStarted","Data":"5ff316bca5c5509582205353959a633ab7094b5c9da1f4269f4c5eaa6302ba8a"} Nov 02 19:56:53.351260 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:53.351165 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" podStartSLOduration=2.321000806 podStartE2EDuration="20.351152904s" totalImagesPullingTime="18.030152098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:53.350265107 +0000 UTC m=+250.158049350" watchObservedRunningTime="2026-11-02 19:56:53.351152904 +0000 UTC m=+250.158937114" Nov 02 19:56:54.247309 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:54.247268 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" event={"ID":"04bc62e6-903c-4f1e-9688-2e4713c7006d","Type":"ContainerStarted","Data":"1ed1ea8bb5b30a6e664bb29ed67f996a0817b3c1ee16b5d9b62610a89f9b22f8"} Nov 02 19:56:54.249254 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:54.249216 5561 generic.go:356] "Generic (PLEG): container finished" podID="fbc884c5-6ca1-40ec-a82f-36815f9173e2" containerID="5ff316bca5c5509582205353959a633ab7094b5c9da1f4269f4c5eaa6302ba8a" exitCode=0 Nov 02 19:56:54.249347 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:54.249269 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" event={"ID":"fbc884c5-6ca1-40ec-a82f-36815f9173e2","Type":"ContainerDied","Data":"5ff316bca5c5509582205353959a633ab7094b5c9da1f4269f4c5eaa6302ba8a"} Nov 02 19:56:54.495501 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:54.495448 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" podStartSLOduration=2.6796063 podStartE2EDuration="21.495433125s" totalImagesPullingTime="18.815826825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:56:54.383626891 +0000 UTC m=+251.191411126" watchObservedRunningTime="2026-11-02 19:56:54.495433125 +0000 UTC m=+251.303217340" Nov 02 19:56:55.259552 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:55.259515 5561 generic.go:356] "Generic (PLEG): container finished" podID="04bc62e6-903c-4f1e-9688-2e4713c7006d" containerID="1ed1ea8bb5b30a6e664bb29ed67f996a0817b3c1ee16b5d9b62610a89f9b22f8" exitCode=0 Nov 02 19:56:55.259966 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:55.259582 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" event={"ID":"04bc62e6-903c-4f1e-9688-2e4713c7006d","Type":"ContainerDied","Data":"1ed1ea8bb5b30a6e664bb29ed67f996a0817b3c1ee16b5d9b62610a89f9b22f8"} Nov 02 19:56:55.411123 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:55.411089 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:55.481854 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:55.481824 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-bundle\") pod \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\" (UID: \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\") " Nov 02 19:56:55.481985 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:55.481882 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc884c5-6ca1-40ec-a82f-36815f9173e2-kube-api-access-tjg8c\" (UniqueName: \"kubernetes.io/projected/fbc884c5-6ca1-40ec-a82f-36815f9173e2-kube-api-access-tjg8c\") pod \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\" (UID: \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\") " Nov 02 19:56:55.481985 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:55.481925 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-util\" (UniqueName: \"kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-util\") pod \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\" (UID: \"fbc884c5-6ca1-40ec-a82f-36815f9173e2\") " Nov 02 19:56:55.482227 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:55.482200 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-bundle" pod "fbc884c5-6ca1-40ec-a82f-36815f9173e2" (UID: "fbc884c5-6ca1-40ec-a82f-36815f9173e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:55.484156 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:55.484132 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc884c5-6ca1-40ec-a82f-36815f9173e2-kube-api-access-tjg8c" pod "fbc884c5-6ca1-40ec-a82f-36815f9173e2" (UID: "fbc884c5-6ca1-40ec-a82f-36815f9173e2"). InnerVolumeSpecName "kube-api-access-tjg8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:56:55.486034 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:55.486011 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-util" pod "fbc884c5-6ca1-40ec-a82f-36815f9173e2" (UID: "fbc884c5-6ca1-40ec-a82f-36815f9173e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:55.582999 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:55.582918 5561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-bundle\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:55.582999 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:55.582951 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjg8c\" (UniqueName: \"kubernetes.io/projected/fbc884c5-6ca1-40ec-a82f-36815f9173e2-kube-api-access-tjg8c\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:55.582999 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:55.582962 5561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc884c5-6ca1-40ec-a82f-36815f9173e2-util\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:56.264860 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.264831 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" Nov 02 19:56:56.264860 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.264849 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sqmg4" event={"ID":"fbc884c5-6ca1-40ec-a82f-36815f9173e2","Type":"ContainerDied","Data":"3c2863e81fe13ec0925e72b852c29bb5c3ea54b2c24b94d849c15708ebdb442e"} Nov 02 19:56:56.265382 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.264908 5561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c2863e81fe13ec0925e72b852c29bb5c3ea54b2c24b94d849c15708ebdb442e" Nov 02 19:56:56.410067 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.410041 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:56:56.490014 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.489981 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-util\" (UniqueName: \"kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-util\") pod \"04bc62e6-903c-4f1e-9688-2e4713c7006d\" (UID: \"04bc62e6-903c-4f1e-9688-2e4713c7006d\") " Nov 02 19:56:56.490165 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.490043 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04bc62e6-903c-4f1e-9688-2e4713c7006d-kube-api-access-m5zfj\" (UniqueName: \"kubernetes.io/projected/04bc62e6-903c-4f1e-9688-2e4713c7006d-kube-api-access-m5zfj\") pod \"04bc62e6-903c-4f1e-9688-2e4713c7006d\" (UID: \"04bc62e6-903c-4f1e-9688-2e4713c7006d\") " Nov 02 19:56:56.490165 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.490068 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-bundle\") pod \"04bc62e6-903c-4f1e-9688-2e4713c7006d\" (UID: \"04bc62e6-903c-4f1e-9688-2e4713c7006d\") " Nov 02 19:56:56.490775 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.490744 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-bundle" pod "04bc62e6-903c-4f1e-9688-2e4713c7006d" (UID: "04bc62e6-903c-4f1e-9688-2e4713c7006d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:56.492241 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.492220 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bc62e6-903c-4f1e-9688-2e4713c7006d-kube-api-access-m5zfj" pod "04bc62e6-903c-4f1e-9688-2e4713c7006d" (UID: "04bc62e6-903c-4f1e-9688-2e4713c7006d"). InnerVolumeSpecName "kube-api-access-m5zfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:56:56.495217 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.495195 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-util" pod "04bc62e6-903c-4f1e-9688-2e4713c7006d" (UID: "04bc62e6-903c-4f1e-9688-2e4713c7006d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:56:56.590969 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.590869 5561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-util\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:56.590969 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.590921 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5zfj\" (UniqueName: \"kubernetes.io/projected/04bc62e6-903c-4f1e-9688-2e4713c7006d-kube-api-access-m5zfj\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:56.590969 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:56.590934 5561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04bc62e6-903c-4f1e-9688-2e4713c7006d-bundle\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:56:57.269896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:57.269851 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" event={"ID":"04bc62e6-903c-4f1e-9688-2e4713c7006d","Type":"ContainerDied","Data":"696d11618e20359a86fb30858ffbfdaf68fd5e495906aca8f18906280e666400"} Nov 02 19:56:57.270271 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:57.269916 5561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="696d11618e20359a86fb30858ffbfdaf68fd5e495906aca8f18906280e666400" Nov 02 19:56:57.270271 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:56:57.269855 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2lb5k" Nov 02 19:57:03.778509 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:57:03.778469 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:57:10.123037 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:10.123003 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:10.305476 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:10.305442 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:12.869498 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:12.869466 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:16.119504 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:16.119468 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:21.094245 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:21.094213 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:22.245979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:22.245945 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:23.726141 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:23.726107 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:26.165417 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:26.165382 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:26.409066 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:26.409029 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:26.616093 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:26.616012 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:26.857310 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:26.857276 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:27.493625 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:27.493587 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:28.341518 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:28.341484 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:28.565051 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:28.565017 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:29.003221 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:29.003186 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:31.287121 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:31.287084 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:31.475523 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:31.475489 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:32.652804 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:32.652766 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:32.861419 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:32.861367 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:33.614755 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.614722 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:33.785124 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:57:33.785082 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:57:33.819560 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.819523 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4"] Nov 02 19:57:33.819984 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.819965 5561 cpu_manager.go:414] "containerMap: removing container" podUID="fbc884c5-6ca1-40ec-a82f-36815f9173e2" containerName="pull" Nov 02 19:57:33.820047 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.819987 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc884c5-6ca1-40ec-a82f-36815f9173e2" containerName="pull" Nov 02 19:57:33.820047 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.820011 5561 cpu_manager.go:414] "containerMap: removing container" podUID="04bc62e6-903c-4f1e-9688-2e4713c7006d" containerName="pull" Nov 02 19:57:33.820047 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.820019 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04bc62e6-903c-4f1e-9688-2e4713c7006d" containerName="pull" Nov 02 19:57:33.820047 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.820032 5561 cpu_manager.go:414] "containerMap: removing container" podUID="fbc884c5-6ca1-40ec-a82f-36815f9173e2" containerName="util" Nov 02 19:57:33.820047 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.820039 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc884c5-6ca1-40ec-a82f-36815f9173e2" containerName="util" Nov 02 19:57:33.820216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.820060 5561 cpu_manager.go:414] "containerMap: removing container" podUID="04bc62e6-903c-4f1e-9688-2e4713c7006d" containerName="util" Nov 02 19:57:33.820216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.820068 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04bc62e6-903c-4f1e-9688-2e4713c7006d" containerName="util" Nov 02 19:57:33.820216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.820081 5561 cpu_manager.go:414] "containerMap: removing container" podUID="fbc884c5-6ca1-40ec-a82f-36815f9173e2" containerName="extract" Nov 02 19:57:33.820216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.820089 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc884c5-6ca1-40ec-a82f-36815f9173e2" containerName="extract" Nov 02 19:57:33.820216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.820098 5561 cpu_manager.go:414] "containerMap: removing container" podUID="04bc62e6-903c-4f1e-9688-2e4713c7006d" containerName="extract" Nov 02 19:57:33.820216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.820105 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04bc62e6-903c-4f1e-9688-2e4713c7006d" containerName="extract" Nov 02 19:57:33.820216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.820180 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc884c5-6ca1-40ec-a82f-36815f9173e2" containerName="extract" Nov 02 19:57:33.820216 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.820191 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="04bc62e6-903c-4f1e-9688-2e4713c7006d" containerName="extract" Nov 02 19:57:33.822588 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.822565 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:33.910279 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.910240 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:33.926957 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.926926 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 19:57:33.927154 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.926996 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 19:57:33.927154 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.927080 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 19:57:33.927246 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.927177 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-km6nn\"" Nov 02 19:57:33.937908 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.937864 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4"] Nov 02 19:57:33.943941 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.943920 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 19:57:33.944049 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.943958 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 19:57:33.990979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.990941 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2c68242-1dcb-4d24-b670-12c65b9c04df-metrics-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:33.991162 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.991020 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2c68242-1dcb-4d24-b670-12c65b9c04df-webhook-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:33.991162 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.991083 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6gbz\" (UniqueName: \"kubernetes.io/projected/a2c68242-1dcb-4d24-b670-12c65b9c04df-kube-api-access-v6gbz\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:33.991162 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.991151 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2c68242-1dcb-4d24-b670-12c65b9c04df-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:33.991352 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.991211 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2c68242-1dcb-4d24-b670-12c65b9c04df-manifests\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:33.991352 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:33.991238 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2c68242-1dcb-4d24-b670-12c65b9c04df-tmp\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.091627 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.091588 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2c68242-1dcb-4d24-b670-12c65b9c04df-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.091627 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.091632 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2c68242-1dcb-4d24-b670-12c65b9c04df-manifests\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.091917 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.091656 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2c68242-1dcb-4d24-b670-12c65b9c04df-tmp\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.091917 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.091689 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2c68242-1dcb-4d24-b670-12c65b9c04df-metrics-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.091917 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.091760 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2c68242-1dcb-4d24-b670-12c65b9c04df-webhook-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.091917 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.091796 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6gbz\" (UniqueName: \"kubernetes.io/projected/a2c68242-1dcb-4d24-b670-12c65b9c04df-kube-api-access-v6gbz\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.092199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.092168 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2c68242-1dcb-4d24-b670-12c65b9c04df-manifests\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.094420 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.094396 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2c68242-1dcb-4d24-b670-12c65b9c04df-tmp\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.094607 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.094405 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2c68242-1dcb-4d24-b670-12c65b9c04df-metrics-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.094854 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.094834 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2c68242-1dcb-4d24-b670-12c65b9c04df-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.095045 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.095018 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2c68242-1dcb-4d24-b670-12c65b9c04df-webhook-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.105815 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.105791 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:34.148789 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.148759 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6gbz\" (UniqueName: \"kubernetes.io/projected/a2c68242-1dcb-4d24-b670-12c65b9c04df-kube-api-access-v6gbz\") pod \"opendatahub-operator-controller-manager-7846bf97f7-25fn4\" (UID: \"a2c68242-1dcb-4d24-b670-12c65b9c04df\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.294381 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.294297 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:34.439892 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.439849 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:34.492295 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.492250 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:34.762123 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.762068 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:34.952254 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:34.952211 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:34.987457 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:57:34.987420 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c68242_1dcb_4d24_b670_12c65b9c04df.slice/crio-3a6ed0e119ea9fa420b55a4ffc91116a53d110b97fe66f3c952cbc2d8db31bdb WatchSource:0}: Error finding container 3a6ed0e119ea9fa420b55a4ffc91116a53d110b97fe66f3c952cbc2d8db31bdb: Status 404 returned error can't find the container with id 3a6ed0e119ea9fa420b55a4ffc91116a53d110b97fe66f3c952cbc2d8db31bdb Nov 02 19:57:35.026335 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:35.026244 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4"] Nov 02 19:57:35.261158 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:35.261110 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:35.397154 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:35.397067 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" event={"ID":"a2c68242-1dcb-4d24-b670-12c65b9c04df","Type":"ContainerStarted","Data":"3a6ed0e119ea9fa420b55a4ffc91116a53d110b97fe66f3c952cbc2d8db31bdb"} Nov 02 19:57:35.802415 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:35.802381 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:36.538857 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.538825 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-hkwvh"] Nov 02 19:57:36.540948 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.540931 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hkwvh" Nov 02 19:57:36.594416 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.594378 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hkwvh"] Nov 02 19:57:36.641210 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.641174 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 19:57:36.641359 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.641228 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ncrdf\"" Nov 02 19:57:36.641359 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.641278 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 19:57:36.714340 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.714300 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd8a1d5b-f681-4208-8221-941c5727877b-bound-sa-token\") pod \"cert-manager-597b95cb84-hkwvh\" (UID: \"fd8a1d5b-f681-4208-8221-941c5727877b\") " pod="cert-manager/cert-manager-597b95cb84-hkwvh" Nov 02 19:57:36.714509 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.714360 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbdb8\" (UniqueName: \"kubernetes.io/projected/fd8a1d5b-f681-4208-8221-941c5727877b-kube-api-access-rbdb8\") pod \"cert-manager-597b95cb84-hkwvh\" (UID: \"fd8a1d5b-f681-4208-8221-941c5727877b\") " pod="cert-manager/cert-manager-597b95cb84-hkwvh" Nov 02 19:57:36.731204 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.731176 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:36.818780 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.818241 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd8a1d5b-f681-4208-8221-941c5727877b-bound-sa-token\") pod \"cert-manager-597b95cb84-hkwvh\" (UID: \"fd8a1d5b-f681-4208-8221-941c5727877b\") " pod="cert-manager/cert-manager-597b95cb84-hkwvh" Nov 02 19:57:36.818780 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.818302 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbdb8\" (UniqueName: \"kubernetes.io/projected/fd8a1d5b-f681-4208-8221-941c5727877b-kube-api-access-rbdb8\") pod \"cert-manager-597b95cb84-hkwvh\" (UID: \"fd8a1d5b-f681-4208-8221-941c5727877b\") " pod="cert-manager/cert-manager-597b95cb84-hkwvh" Nov 02 19:57:36.873637 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.873605 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd8a1d5b-f681-4208-8221-941c5727877b-bound-sa-token\") pod \"cert-manager-597b95cb84-hkwvh\" (UID: \"fd8a1d5b-f681-4208-8221-941c5727877b\") " pod="cert-manager/cert-manager-597b95cb84-hkwvh" Nov 02 19:57:36.873803 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:36.873706 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbdb8\" (UniqueName: \"kubernetes.io/projected/fd8a1d5b-f681-4208-8221-941c5727877b-kube-api-access-rbdb8\") pod \"cert-manager-597b95cb84-hkwvh\" (UID: \"fd8a1d5b-f681-4208-8221-941c5727877b\") " pod="cert-manager/cert-manager-597b95cb84-hkwvh" Nov 02 19:57:37.163616 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:37.163573 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hkwvh" Nov 02 19:57:37.530378 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:37.530286 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hkwvh"] Nov 02 19:57:37.538399 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:57:37.538370 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd8a1d5b_f681_4208_8221_941c5727877b.slice/crio-a91863c50a2e0b6b5ef37c76b94eacc951c377308c3eb88da87b3849994033cf WatchSource:0}: Error finding container a91863c50a2e0b6b5ef37c76b94eacc951c377308c3eb88da87b3849994033cf: Status 404 returned error can't find the container with id a91863c50a2e0b6b5ef37c76b94eacc951c377308c3eb88da87b3849994033cf Nov 02 19:57:38.292158 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:38.292117 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:38.412149 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:38.412112 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hkwvh" event={"ID":"fd8a1d5b-f681-4208-8221-941c5727877b","Type":"ContainerStarted","Data":"a91863c50a2e0b6b5ef37c76b94eacc951c377308c3eb88da87b3849994033cf"} Nov 02 19:57:38.698125 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:38.698089 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:39.417993 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:39.417955 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" event={"ID":"a2c68242-1dcb-4d24-b670-12c65b9c04df","Type":"ContainerStarted","Data":"09b4c5d74b3cb544da8ce194e4d298edb641c675e2b4e1a46a82315a6cf582b7"} Nov 02 19:57:40.424242 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:40.424210 5561 generic.go:356] "Generic (PLEG): container finished" podID="a2c68242-1dcb-4d24-b670-12c65b9c04df" containerID="09b4c5d74b3cb544da8ce194e4d298edb641c675e2b4e1a46a82315a6cf582b7" exitCode=0 Nov 02 19:57:40.424676 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:40.424296 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" event={"ID":"a2c68242-1dcb-4d24-b670-12c65b9c04df","Type":"ContainerDied","Data":"09b4c5d74b3cb544da8ce194e4d298edb641c675e2b4e1a46a82315a6cf582b7"} Nov 02 19:57:40.493917 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:40.493865 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:41.431756 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:41.431689 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" event={"ID":"a2c68242-1dcb-4d24-b670-12c65b9c04df","Type":"ContainerStarted","Data":"e638cc234bcf21f771108e0ffc252a8730d72bfb71a057b359061547874dda79"} Nov 02 19:57:41.432262 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:41.431853 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:41.557801 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:41.557702 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" podStartSLOduration=2.694579806 podStartE2EDuration="8.557689794s" totalImagesPullingTime="5.200456606s" totalInitContainerRuntime="662.653382ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:41.556613037 +0000 UTC m=+298.364397291" watchObservedRunningTime="2026-11-02 19:57:41.557689794 +0000 UTC m=+298.365474009" Nov 02 19:57:42.437129 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:42.437096 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hkwvh" event={"ID":"fd8a1d5b-f681-4208-8221-941c5727877b","Type":"ContainerStarted","Data":"01b990a2ca58c3df12b1c145c0bff69a37987da679f4454ebe87abd7ae3aaa25"} Nov 02 19:57:42.467560 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:42.467531 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:42.547421 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:42.547369 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-hkwvh" podStartSLOduration=2.595210014 podStartE2EDuration="6.547356375s" totalImagesPullingTime="3.952146361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:42.546796755 +0000 UTC m=+299.354580972" watchObservedRunningTime="2026-11-02 19:57:42.547356375 +0000 UTC m=+299.355140588" Nov 02 19:57:43.283245 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:43.283213 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:43.472541 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:43.472512 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:43.851301 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:43.851263 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:44.039940 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:44.039906 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:48.007548 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:48.007516 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:48.247540 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:48.247505 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:48.430485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:48.430450 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:48.648094 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:48.648044 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:48.823552 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:48.823462 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:49.016102 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:49.016066 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:52.439960 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:52.439931 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-25fn4" Nov 02 19:57:54.542473 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:54.542411 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:54.750404 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:54.750361 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:59.247896 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:59.247842 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:57:59.727844 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:57:59.727808 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:03.030531 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:03.030497 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:03.206050 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:03.206005 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:03.616827 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:03.616788 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:03.778121 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:58:03.778082 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:58:03.792833 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:03.792805 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:04.874324 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:04.874286 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:05.076006 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:05.075973 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:05.338033 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:05.338001 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:06.085520 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:06.080597 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:06.383496 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:06.383415 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:07.086087 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.086053 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:07.091826 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.091800 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67b88d44-vp4mm"] Nov 02 19:58:07.094622 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.094602 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" Nov 02 19:58:07.147755 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.147716 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67b88d44-vp4mm"] Nov 02 19:58:07.157851 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.157814 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 19:58:07.165841 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.165811 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aaf3106-a6bd-45eb-aa93-1f6a257afdc2-tls-certs\") pod \"kube-auth-proxy-67b88d44-vp4mm\" (UID: \"5aaf3106-a6bd-45eb-aa93-1f6a257afdc2\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" Nov 02 19:58:07.165984 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.165851 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l59nm\" (UniqueName: \"kubernetes.io/projected/5aaf3106-a6bd-45eb-aa93-1f6a257afdc2-kube-api-access-l59nm\") pod \"kube-auth-proxy-67b88d44-vp4mm\" (UID: \"5aaf3106-a6bd-45eb-aa93-1f6a257afdc2\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" Nov 02 19:58:07.165984 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.165936 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5aaf3106-a6bd-45eb-aa93-1f6a257afdc2-tmp\") pod \"kube-auth-proxy-67b88d44-vp4mm\" (UID: \"5aaf3106-a6bd-45eb-aa93-1f6a257afdc2\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" Nov 02 19:58:07.193858 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.193831 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cjjfn\"" Nov 02 19:58:07.194020 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.193894 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 19:58:07.267112 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.267078 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5aaf3106-a6bd-45eb-aa93-1f6a257afdc2-tmp\") pod \"kube-auth-proxy-67b88d44-vp4mm\" (UID: \"5aaf3106-a6bd-45eb-aa93-1f6a257afdc2\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" Nov 02 19:58:07.267288 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.267131 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aaf3106-a6bd-45eb-aa93-1f6a257afdc2-tls-certs\") pod \"kube-auth-proxy-67b88d44-vp4mm\" (UID: \"5aaf3106-a6bd-45eb-aa93-1f6a257afdc2\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" Nov 02 19:58:07.267288 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.267159 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59nm\" (UniqueName: \"kubernetes.io/projected/5aaf3106-a6bd-45eb-aa93-1f6a257afdc2-kube-api-access-l59nm\") pod \"kube-auth-proxy-67b88d44-vp4mm\" (UID: \"5aaf3106-a6bd-45eb-aa93-1f6a257afdc2\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" Nov 02 19:58:07.269297 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.269270 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5aaf3106-a6bd-45eb-aa93-1f6a257afdc2-tmp\") pod \"kube-auth-proxy-67b88d44-vp4mm\" (UID: \"5aaf3106-a6bd-45eb-aa93-1f6a257afdc2\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" Nov 02 19:58:07.269552 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.269532 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aaf3106-a6bd-45eb-aa93-1f6a257afdc2-tls-certs\") pod \"kube-auth-proxy-67b88d44-vp4mm\" (UID: \"5aaf3106-a6bd-45eb-aa93-1f6a257afdc2\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" Nov 02 19:58:07.282206 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.282169 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:07.320673 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.320648 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59nm\" (UniqueName: \"kubernetes.io/projected/5aaf3106-a6bd-45eb-aa93-1f6a257afdc2-kube-api-access-l59nm\") pod \"kube-auth-proxy-67b88d44-vp4mm\" (UID: \"5aaf3106-a6bd-45eb-aa93-1f6a257afdc2\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" Nov 02 19:58:07.404613 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.404585 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" Nov 02 19:58:07.465544 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.465515 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:07.676902 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.676819 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:07.771206 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.771171 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67b88d44-vp4mm"] Nov 02 19:58:07.779801 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:58:07.779758 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aaf3106_a6bd_45eb_aa93_1f6a257afdc2.slice/crio-44be4c1a8c1ba8e8b81b12c158a103adefa5476eae0fd35a7dc25f5f91a2d185 WatchSource:0}: Error finding container 44be4c1a8c1ba8e8b81b12c158a103adefa5476eae0fd35a7dc25f5f91a2d185: Status 404 returned error can't find the container with id 44be4c1a8c1ba8e8b81b12c158a103adefa5476eae0fd35a7dc25f5f91a2d185 Nov 02 19:58:07.859604 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:07.859562 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:08.046215 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:08.046137 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:08.435563 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:08.435532 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:08.534156 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:08.534121 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" event={"ID":"5aaf3106-a6bd-45eb-aa93-1f6a257afdc2","Type":"ContainerStarted","Data":"44be4c1a8c1ba8e8b81b12c158a103adefa5476eae0fd35a7dc25f5f91a2d185"} Nov 02 19:58:08.604135 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:08.604099 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:08.838173 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:08.838089 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:09.038580 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:09.038550 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:09.681661 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:09.681622 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:09.929735 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:09.929707 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:10.103563 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.103496 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:10.318792 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.318754 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk"] Nov 02 19:58:10.330446 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.330417 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.374539 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.374460 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk"] Nov 02 19:58:10.400664 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.400618 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnmln\" (UniqueName: \"kubernetes.io/projected/6d2592fd-465d-4d4e-bc6f-8957342f8f39-kube-api-access-nnmln\") pod \"jobset-controller-manager-65f56c849f-hsrrk\" (UID: \"6d2592fd-465d-4d4e-bc6f-8957342f8f39\") " pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.400854 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.400717 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d2592fd-465d-4d4e-bc6f-8957342f8f39-cert\") pod \"jobset-controller-manager-65f56c849f-hsrrk\" (UID: \"6d2592fd-465d-4d4e-bc6f-8957342f8f39\") " pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.400949 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.400847 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6d2592fd-465d-4d4e-bc6f-8957342f8f39-manager-config\") pod \"jobset-controller-manager-65f56c849f-hsrrk\" (UID: \"6d2592fd-465d-4d4e-bc6f-8957342f8f39\") " pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.400949 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.400900 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2592fd-465d-4d4e-bc6f-8957342f8f39-metrics-certs\") pod \"jobset-controller-manager-65f56c849f-hsrrk\" (UID: \"6d2592fd-465d-4d4e-bc6f-8957342f8f39\") " pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.421217 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.421190 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 19:58:10.427610 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.427565 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:58:10.427610 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.427599 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 19:58:10.427832 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.427638 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 19:58:10.427832 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.427702 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-fr6dx\"" Nov 02 19:58:10.428135 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.428109 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 19:58:10.501361 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.501323 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnmln\" (UniqueName: \"kubernetes.io/projected/6d2592fd-465d-4d4e-bc6f-8957342f8f39-kube-api-access-nnmln\") pod \"jobset-controller-manager-65f56c849f-hsrrk\" (UID: \"6d2592fd-465d-4d4e-bc6f-8957342f8f39\") " pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.501551 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.501375 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d2592fd-465d-4d4e-bc6f-8957342f8f39-cert\") pod \"jobset-controller-manager-65f56c849f-hsrrk\" (UID: \"6d2592fd-465d-4d4e-bc6f-8957342f8f39\") " pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.501551 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.501439 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6d2592fd-465d-4d4e-bc6f-8957342f8f39-manager-config\") pod \"jobset-controller-manager-65f56c849f-hsrrk\" (UID: \"6d2592fd-465d-4d4e-bc6f-8957342f8f39\") " pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.501551 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.501470 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2592fd-465d-4d4e-bc6f-8957342f8f39-metrics-certs\") pod \"jobset-controller-manager-65f56c849f-hsrrk\" (UID: \"6d2592fd-465d-4d4e-bc6f-8957342f8f39\") " pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.502364 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.502339 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6d2592fd-465d-4d4e-bc6f-8957342f8f39-manager-config\") pod \"jobset-controller-manager-65f56c849f-hsrrk\" (UID: \"6d2592fd-465d-4d4e-bc6f-8957342f8f39\") " pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.504272 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.504250 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d2592fd-465d-4d4e-bc6f-8957342f8f39-cert\") pod \"jobset-controller-manager-65f56c849f-hsrrk\" (UID: \"6d2592fd-465d-4d4e-bc6f-8957342f8f39\") " pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.504401 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.504291 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2592fd-465d-4d4e-bc6f-8957342f8f39-metrics-certs\") pod \"jobset-controller-manager-65f56c849f-hsrrk\" (UID: \"6d2592fd-465d-4d4e-bc6f-8957342f8f39\") " pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.556709 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.556673 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnmln\" (UniqueName: \"kubernetes.io/projected/6d2592fd-465d-4d4e-bc6f-8957342f8f39-kube-api-access-nnmln\") pod \"jobset-controller-manager-65f56c849f-hsrrk\" (UID: \"6d2592fd-465d-4d4e-bc6f-8957342f8f39\") " pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:10.646312 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:10.646217 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:11.022667 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:11.022634 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk"] Nov 02 19:58:11.071855 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:58:11.071809 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2592fd_465d_4d4e_bc6f_8957342f8f39.slice/crio-d686e7cb9c1c8970fa60750f820bde9dd8f801db930fe463306f57427a752a17 WatchSource:0}: Error finding container d686e7cb9c1c8970fa60750f820bde9dd8f801db930fe463306f57427a752a17: Status 404 returned error can't find the container with id d686e7cb9c1c8970fa60750f820bde9dd8f801db930fe463306f57427a752a17 Nov 02 19:58:11.251950 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:11.251902 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:11.428540 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:11.428495 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:11.555371 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:11.555329 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" event={"ID":"5aaf3106-a6bd-45eb-aa93-1f6a257afdc2","Type":"ContainerStarted","Data":"6584a9edd0d26249b44ba16b2c7ec30096ae46fd17493053e3387e28bbcbe615"} Nov 02 19:58:11.556589 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:11.556552 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" event={"ID":"6d2592fd-465d-4d4e-bc6f-8957342f8f39","Type":"ContainerStarted","Data":"d686e7cb9c1c8970fa60750f820bde9dd8f801db930fe463306f57427a752a17"} Nov 02 19:58:11.664603 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:11.664546 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67b88d44-vp4mm" podStartSLOduration=1.046457785 podStartE2EDuration="4.664531702s" totalImagesPullingTime="3.618073917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:11.663606682 +0000 UTC m=+328.471390916" watchObservedRunningTime="2026-11-02 19:58:11.664531702 +0000 UTC m=+328.472315918" Nov 02 19:58:14.211406 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:14.211359 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:14.388948 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:14.388915 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:15.578676 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:15.578594 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" event={"ID":"6d2592fd-465d-4d4e-bc6f-8957342f8f39","Type":"ContainerStarted","Data":"858c45e858ae4fb2aa63feead9137d555baf1a3819d1fb9677be18547a992b7b"} Nov 02 19:58:15.579035 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:15.578741 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:15.686432 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:15.686386 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" podStartSLOduration=1.5703403379999998 podStartE2EDuration="5.686371649s" totalImagesPullingTime="4.116031311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:15.686060617 +0000 UTC m=+332.493844854" watchObservedRunningTime="2026-11-02 19:58:15.686371649 +0000 UTC m=+332.494155865" Nov 02 19:58:19.026985 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:19.026948 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:19.200220 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:19.200150 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:26.588460 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:26.588428 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-65f56c849f-hsrrk" Nov 02 19:58:32.258233 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:32.258200 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:32.472695 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:32.472610 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:33.034092 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:33.034058 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:33.778498 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:58:33.778463 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:58:42.803656 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:42.803621 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:52.531337 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:52.531302 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:52.702968 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:52.702933 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:54.042235 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:54.042157 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:54.223377 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:54.223341 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:54.403250 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:54.403216 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:54.602202 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:54.602166 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:58.979018 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:58.978982 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:58:59.152757 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:58:59.152714 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:01.824373 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:01.824340 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:02.119235 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:02.119139 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:03.778314 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:59:03.778276 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:59:24.823944 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:24.823909 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:25.193428 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:25.193389 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:25.373950 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:25.373915 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:25.602747 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:25.602646 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:25.804374 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:25.804330 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:25.974008 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:25.973974 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:26.000206 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.000172 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-xj6qf"] Nov 02 19:59:26.003118 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.003096 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:26.056553 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.056520 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-xj6qf"] Nov 02 19:59:26.103156 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.103115 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 19:59:26.103156 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.103139 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 19:59:26.103156 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.103122 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 19:59:26.103156 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.103122 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 19:59:26.103440 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.103204 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-td8k6\"" Nov 02 19:59:26.128240 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.128209 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebc020a9-bd01-4f50-be4d-d8ff574f6c75-cert\") pod \"odh-model-controller-769875db56-xj6qf\" (UID: \"ebc020a9-bd01-4f50-be4d-d8ff574f6c75\") " pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:26.128376 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.128246 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxgzs\" (UniqueName: \"kubernetes.io/projected/ebc020a9-bd01-4f50-be4d-d8ff574f6c75-kube-api-access-wxgzs\") pod \"odh-model-controller-769875db56-xj6qf\" (UID: \"ebc020a9-bd01-4f50-be4d-d8ff574f6c75\") " pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:26.128376 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.128332 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebc020a9-bd01-4f50-be4d-d8ff574f6c75-metrics-certs\") pod \"odh-model-controller-769875db56-xj6qf\" (UID: \"ebc020a9-bd01-4f50-be4d-d8ff574f6c75\") " pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:26.229025 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.228937 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebc020a9-bd01-4f50-be4d-d8ff574f6c75-cert\") pod \"odh-model-controller-769875db56-xj6qf\" (UID: \"ebc020a9-bd01-4f50-be4d-d8ff574f6c75\") " pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:26.229025 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.228979 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxgzs\" (UniqueName: \"kubernetes.io/projected/ebc020a9-bd01-4f50-be4d-d8ff574f6c75-kube-api-access-wxgzs\") pod \"odh-model-controller-769875db56-xj6qf\" (UID: \"ebc020a9-bd01-4f50-be4d-d8ff574f6c75\") " pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:26.229252 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.229229 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebc020a9-bd01-4f50-be4d-d8ff574f6c75-metrics-certs\") pod \"odh-model-controller-769875db56-xj6qf\" (UID: \"ebc020a9-bd01-4f50-be4d-d8ff574f6c75\") " pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:26.231568 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.231547 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebc020a9-bd01-4f50-be4d-d8ff574f6c75-metrics-certs\") pod \"odh-model-controller-769875db56-xj6qf\" (UID: \"ebc020a9-bd01-4f50-be4d-d8ff574f6c75\") " pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:26.231568 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.231550 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebc020a9-bd01-4f50-be4d-d8ff574f6c75-cert\") pod \"odh-model-controller-769875db56-xj6qf\" (UID: \"ebc020a9-bd01-4f50-be4d-d8ff574f6c75\") " pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:26.282848 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.282814 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxgzs\" (UniqueName: \"kubernetes.io/projected/ebc020a9-bd01-4f50-be4d-d8ff574f6c75-kube-api-access-wxgzs\") pod \"odh-model-controller-769875db56-xj6qf\" (UID: \"ebc020a9-bd01-4f50-be4d-d8ff574f6c75\") " pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:26.316379 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.316338 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:26.681796 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.681750 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-xj6qf"] Nov 02 19:59:26.690098 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:59:26.690059 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc020a9_bd01_4f50_be4d_d8ff574f6c75.slice/crio-191959611ed1e27a25fb6794b14a38ab728e1779cd62c72daf899f7bd340974d WatchSource:0}: Error finding container 191959611ed1e27a25fb6794b14a38ab728e1779cd62c72daf899f7bd340974d: Status 404 returned error can't find the container with id 191959611ed1e27a25fb6794b14a38ab728e1779cd62c72daf899f7bd340974d Nov 02 19:59:26.875785 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:26.875751 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-xj6qf" event={"ID":"ebc020a9-bd01-4f50-be4d-d8ff574f6c75","Type":"ContainerStarted","Data":"191959611ed1e27a25fb6794b14a38ab728e1779cd62c72daf899f7bd340974d"} Nov 02 19:59:27.095979 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:27.095903 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:27.531329 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:27.531294 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:27.707193 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:27.707162 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:29.615129 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:29.615094 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:29.851337 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:29.851304 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:30.017230 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.017185 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:30.579044 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.578969 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b"] Nov 02 19:59:30.579196 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.579170 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" podUID="639bf58d-c6c3-41ec-a22f-939b92e0b24e" containerName="discovery" containerID="cri-o://6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c" gracePeriod=30 Nov 02 19:59:30.692460 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.692439 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r"] Nov 02 19:59:30.695685 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.695666 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:30.748734 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.748707 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 19:59:30.748869 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.748764 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 19:59:30.748869 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.748800 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vqtwv\"" Nov 02 19:59:30.748869 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.748862 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 19:59:30.749548 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.749502 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 19:59:30.750259 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.750196 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r"] Nov 02 19:59:30.822466 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.822440 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:59:30.872079 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.872000 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def7ada1-4544-4fab-800e-8375b1d1983a-openshift-service-ca\") pod \"maas-api-6d669c7fcd-t9h7r\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:30.872079 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.872053 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-metrics-certs\") pod \"maas-api-6d669c7fcd-t9h7r\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:30.872246 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.872131 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-maas-api-tls\") pod \"maas-api-6d669c7fcd-t9h7r\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:30.872246 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.872213 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4748\" (UniqueName: \"kubernetes.io/projected/def7ada1-4544-4fab-800e-8375b1d1983a-kube-api-access-r4748\") pod \"maas-api-6d669c7fcd-t9h7r\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:30.897063 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.897026 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-xj6qf" event={"ID":"ebc020a9-bd01-4f50-be4d-d8ff574f6c75","Type":"ContainerStarted","Data":"ef7eeec184bfab6843d6888b444bd768448745d033032e3468c00dd26032b347"} Nov 02 19:59:30.897236 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.897096 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:30.898393 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.898368 5561 generic.go:356] "Generic (PLEG): container finished" podID="639bf58d-c6c3-41ec-a22f-939b92e0b24e" containerID="6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c" exitCode=0 Nov 02 19:59:30.898511 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.898393 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" event={"ID":"639bf58d-c6c3-41ec-a22f-939b92e0b24e","Type":"ContainerDied","Data":"6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c"} Nov 02 19:59:30.898511 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.898407 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" Nov 02 19:59:30.898511 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.898435 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b" event={"ID":"639bf58d-c6c3-41ec-a22f-939b92e0b24e","Type":"ContainerDied","Data":"05aad7b22e9bd3f010f6ff4665dc519c06316e33fae5451f6a7edf3333807c09"} Nov 02 19:59:30.898511 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.898454 5561 scope.go:122] "RemoveContainer" containerID="6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c" Nov 02 19:59:30.907090 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.907073 5561 scope.go:122] "RemoveContainer" containerID="6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c" Nov 02 19:59:30.907369 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:59:30.907353 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c\": container with ID starting with 6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c not found: ID does not exist" containerID="6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c" Nov 02 19:59:30.907422 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.907375 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c"} err="failed to get container status \"6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c\": rpc error: code = NotFound desc = could not find container \"6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c\": container with ID starting with 6af8b84c42a930cae50a316ecfa8156a80f6f0c2a5b5eebbbcf39df74968d37c not found: ID does not exist" Nov 02 19:59:30.973493 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.973462 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-token\" (UniqueName: \"kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-token\") pod \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " Nov 02 19:59:30.973647 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.973503 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-dns-cert\") pod \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " Nov 02 19:59:30.973647 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.973554 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-kubeconfig\") pod \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " Nov 02 19:59:30.973647 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.973586 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-kube-api-access-nvj6x\" (UniqueName: \"kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-kube-api-access-nvj6x\") pod \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " Nov 02 19:59:30.973820 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.973658 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639bf58d-c6c3-41ec-a22f-939b92e0b24e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/639bf58d-c6c3-41ec-a22f-939b92e0b24e-local-certs\") pod \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " Nov 02 19:59:30.973820 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.973716 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-cacerts\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-cacerts\") pod \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " Nov 02 19:59:30.973820 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.973749 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-ca-configmap\") pod \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\" (UID: \"639bf58d-c6c3-41ec-a22f-939b92e0b24e\") " Nov 02 19:59:30.974028 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.973864 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def7ada1-4544-4fab-800e-8375b1d1983a-openshift-service-ca\") pod \"maas-api-6d669c7fcd-t9h7r\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:30.974028 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.973937 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-metrics-certs\") pod \"maas-api-6d669c7fcd-t9h7r\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:30.974028 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.973981 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-maas-api-tls\") pod \"maas-api-6d669c7fcd-t9h7r\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:30.974184 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.974047 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4748\" (UniqueName: \"kubernetes.io/projected/def7ada1-4544-4fab-800e-8375b1d1983a-kube-api-access-r4748\") pod \"maas-api-6d669c7fcd-t9h7r\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:30.974260 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.974225 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-ca-configmap" pod "639bf58d-c6c3-41ec-a22f-939b92e0b24e" (UID: "639bf58d-c6c3-41ec-a22f-939b92e0b24e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:59:30.975236 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.975207 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def7ada1-4544-4fab-800e-8375b1d1983a-openshift-service-ca\") pod \"maas-api-6d669c7fcd-t9h7r\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:30.976348 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.976318 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-dns-cert" pod "639bf58d-c6c3-41ec-a22f-939b92e0b24e" (UID: "639bf58d-c6c3-41ec-a22f-939b92e0b24e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:59:30.976456 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.976327 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-token" pod "639bf58d-c6c3-41ec-a22f-939b92e0b24e" (UID: "639bf58d-c6c3-41ec-a22f-939b92e0b24e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:59:30.976534 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.976475 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639bf58d-c6c3-41ec-a22f-939b92e0b24e-local-certs" pod "639bf58d-c6c3-41ec-a22f-939b92e0b24e" (UID: "639bf58d-c6c3-41ec-a22f-939b92e0b24e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:59:30.976844 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.976808 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-kube-api-access-nvj6x" pod "639bf58d-c6c3-41ec-a22f-939b92e0b24e" (UID: "639bf58d-c6c3-41ec-a22f-939b92e0b24e"). InnerVolumeSpecName "kube-api-access-nvj6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:59:30.976984 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.976899 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-cacerts" pod "639bf58d-c6c3-41ec-a22f-939b92e0b24e" (UID: "639bf58d-c6c3-41ec-a22f-939b92e0b24e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:59:30.977199 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.977176 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-metrics-certs\") pod \"maas-api-6d669c7fcd-t9h7r\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:30.977315 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.977296 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-maas-api-tls\") pod \"maas-api-6d669c7fcd-t9h7r\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:30.977604 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:30.977587 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-kubeconfig" pod "639bf58d-c6c3-41ec-a22f-939b92e0b24e" (UID: "639bf58d-c6c3-41ec-a22f-939b92e0b24e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:59:31.028247 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.028215 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4748\" (UniqueName: \"kubernetes.io/projected/def7ada1-4544-4fab-800e-8375b1d1983a-kube-api-access-r4748\") pod \"maas-api-6d669c7fcd-t9h7r\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:31.033192 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.033169 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:31.044169 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.044120 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-xj6qf" podStartSLOduration=2.463790789 podStartE2EDuration="6.044106829s" totalImagesPullingTime="3.58031604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:31.043846497 +0000 UTC m=+407.851630732" watchObservedRunningTime="2026-11-02 19:59:31.044106829 +0000 UTC m=+407.851891092" Nov 02 19:59:31.075286 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.075251 5561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/639bf58d-c6c3-41ec-a22f-939b92e0b24e-local-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:31.075440 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.075348 5561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-cacerts\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:31.075440 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.075362 5561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-ca-configmap\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:31.075440 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.075371 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:31.075440 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.075380 5561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-csr-dns-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:31.075440 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.075389 5561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/639bf58d-c6c3-41ec-a22f-939b92e0b24e-istio-kubeconfig\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:31.075440 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.075397 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvj6x\" (UniqueName: \"kubernetes.io/projected/639bf58d-c6c3-41ec-a22f-939b92e0b24e-kube-api-access-nvj6x\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:31.312849 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.312819 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b"] Nov 02 19:59:31.318541 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.318515 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mkg8b"] Nov 02 19:59:31.429589 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.429552 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r"] Nov 02 19:59:31.437914 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:59:31.437885 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef7ada1_4544_4fab_800e_8375b1d1983a.slice/crio-2c31846090cb6027cfdc9985dad7cd39873d6230b8410f75e0841a1f418ce851 WatchSource:0}: Error finding container 2c31846090cb6027cfdc9985dad7cd39873d6230b8410f75e0841a1f418ce851: Status 404 returned error can't find the container with id 2c31846090cb6027cfdc9985dad7cd39873d6230b8410f75e0841a1f418ce851 Nov 02 19:59:31.903497 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:31.903463 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" event={"ID":"def7ada1-4544-4fab-800e-8375b1d1983a","Type":"ContainerStarted","Data":"2c31846090cb6027cfdc9985dad7cd39873d6230b8410f75e0841a1f418ce851"} Nov 02 19:59:32.097092 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:32.097061 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:32.271091 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:32.271010 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:32.448536 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:32.448487 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:32.728538 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:32.728502 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:32.903425 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:32.903390 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:33.082283 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:33.082203 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:33.256504 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:33.256464 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:33.313190 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:33.313156 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="639bf58d-c6c3-41ec-a22f-939b92e0b24e" path="/var/lib/kubelet/pods/639bf58d-c6c3-41ec-a22f-939b92e0b24e/volumes" Nov 02 19:59:33.425868 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:33.425836 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:33.598485 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:33.598421 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:33.783824 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:59:33.783725 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 19:59:34.918047 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:34.918010 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" event={"ID":"def7ada1-4544-4fab-800e-8375b1d1983a","Type":"ContainerStarted","Data":"0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5"} Nov 02 19:59:34.918474 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:34.918240 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:35.027846 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:35.027791 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" podStartSLOduration=1.8385802070000001 podStartE2EDuration="5.027777595s" totalImagesPullingTime="3.189197388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:35.02662275 +0000 UTC m=+411.834406987" watchObservedRunningTime="2026-11-02 19:59:35.027777595 +0000 UTC m=+411.835561810" Nov 02 19:59:40.928271 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:40.928241 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:41.468387 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:41.468348 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:41.755300 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:41.755218 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:41.886702 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:41.886667 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf"] Nov 02 19:59:41.887024 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:41.887000 5561 cpu_manager.go:414] "containerMap: removing container" podUID="639bf58d-c6c3-41ec-a22f-939b92e0b24e" containerName="discovery" Nov 02 19:59:41.887024 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:41.887025 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639bf58d-c6c3-41ec-a22f-939b92e0b24e" containerName="discovery" Nov 02 19:59:41.887137 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:41.887081 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="639bf58d-c6c3-41ec-a22f-939b92e0b24e" containerName="discovery" Nov 02 19:59:41.890112 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:41.890091 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:41.906765 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:41.906745 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-xj6qf" Nov 02 19:59:41.989047 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:41.989015 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf"] Nov 02 19:59:42.067607 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.067507 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-metrics-certs\") pod \"maas-api-556d9884fd-cdbxf\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.067759 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.067694 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-maas-api-tls\") pod \"maas-api-556d9884fd-cdbxf\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.067817 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.067792 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjmvs\" (UniqueName: \"kubernetes.io/projected/5a7307c1-b1e6-4858-9f67-f9b8022f5279-kube-api-access-mjmvs\") pod \"maas-api-556d9884fd-cdbxf\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.067895 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.067851 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7307c1-b1e6-4858-9f67-f9b8022f5279-openshift-service-ca\") pod \"maas-api-556d9884fd-cdbxf\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.168843 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.168811 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-maas-api-tls\") pod \"maas-api-556d9884fd-cdbxf\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.169015 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.168862 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmvs\" (UniqueName: \"kubernetes.io/projected/5a7307c1-b1e6-4858-9f67-f9b8022f5279-kube-api-access-mjmvs\") pod \"maas-api-556d9884fd-cdbxf\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.169015 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.168919 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7307c1-b1e6-4858-9f67-f9b8022f5279-openshift-service-ca\") pod \"maas-api-556d9884fd-cdbxf\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.169015 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.168955 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-metrics-certs\") pod \"maas-api-556d9884fd-cdbxf\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.169627 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.169604 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7307c1-b1e6-4858-9f67-f9b8022f5279-openshift-service-ca\") pod \"maas-api-556d9884fd-cdbxf\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.171284 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.171263 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-maas-api-tls\") pod \"maas-api-556d9884fd-cdbxf\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.171375 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.171303 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-metrics-certs\") pod \"maas-api-556d9884fd-cdbxf\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.223015 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.222988 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjmvs\" (UniqueName: \"kubernetes.io/projected/5a7307c1-b1e6-4858-9f67-f9b8022f5279-kube-api-access-mjmvs\") pod \"maas-api-556d9884fd-cdbxf\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.502273 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.502237 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:42.926048 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.926016 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf"] Nov 02 19:59:42.935030 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:59:42.934997 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7307c1_b1e6_4858_9f67_f9b8022f5279.slice/crio-f417351c2ee73e7520860e772a9a81066d92bc5f98cbdc3ae205802d87b3b60b WatchSource:0}: Error finding container f417351c2ee73e7520860e772a9a81066d92bc5f98cbdc3ae205802d87b3b60b: Status 404 returned error can't find the container with id f417351c2ee73e7520860e772a9a81066d92bc5f98cbdc3ae205802d87b3b60b Nov 02 19:59:42.957214 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:42.957183 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" event={"ID":"5a7307c1-b1e6-4858-9f67-f9b8022f5279","Type":"ContainerStarted","Data":"f417351c2ee73e7520860e772a9a81066d92bc5f98cbdc3ae205802d87b3b60b"} Nov 02 19:59:43.048860 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.048828 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-586f56887b-fzjpt"] Nov 02 19:59:43.053367 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.053343 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586f56887b-fzjpt" Nov 02 19:59:43.099737 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.099709 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:43.103253 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.103227 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-586f56887b-fzjpt"] Nov 02 19:59:43.103486 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.103464 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 19:59:43.103581 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.103520 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 19:59:43.149859 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.149828 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5v79g\"" Nov 02 19:59:43.177636 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.177569 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2csn\" (UniqueName: \"kubernetes.io/projected/e32544f6-220d-47a8-8155-518c53a314fe-kube-api-access-h2csn\") pod \"authorino-586f56887b-fzjpt\" (UID: \"e32544f6-220d-47a8-8155-518c53a314fe\") " pod="kuadrant-system/authorino-586f56887b-fzjpt" Nov 02 19:59:43.277953 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.277915 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2csn\" (UniqueName: \"kubernetes.io/projected/e32544f6-220d-47a8-8155-518c53a314fe-kube-api-access-h2csn\") pod \"authorino-586f56887b-fzjpt\" (UID: \"e32544f6-220d-47a8-8155-518c53a314fe\") " pod="kuadrant-system/authorino-586f56887b-fzjpt" Nov 02 19:59:43.376343 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.376315 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 19:59:43.379083 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.379069 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 19:59:43.429625 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.429566 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 19:59:43.433882 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.433852 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2csn\" (UniqueName: \"kubernetes.io/projected/e32544f6-220d-47a8-8155-518c53a314fe-kube-api-access-h2csn\") pod \"authorino-586f56887b-fzjpt\" (UID: \"e32544f6-220d-47a8-8155-518c53a314fe\") " pod="kuadrant-system/authorino-586f56887b-fzjpt" Nov 02 19:59:43.714065 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.714046 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5v79g\"" Nov 02 19:59:43.716145 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.716126 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586f56887b-fzjpt" Nov 02 19:59:43.962898 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.962798 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" event={"ID":"5a7307c1-b1e6-4858-9f67-f9b8022f5279","Type":"ContainerStarted","Data":"65ea736a06a51e434d299c537d1c60564c80b0a6bd5ce15b512681571d4aebf9"} Nov 02 19:59:43.962898 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:43.962895 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:44.090688 ip-10-0-134-194 kubenswrapper[5561]: W1102 19:59:44.090654 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode32544f6_220d_47a8_8155_518c53a314fe.slice/crio-622efcbefb679a0554cb2934c85a67fc5720afed24cf196a171fdfe4c459061f WatchSource:0}: Error finding container 622efcbefb679a0554cb2934c85a67fc5720afed24cf196a171fdfe4c459061f: Status 404 returned error can't find the container with id 622efcbefb679a0554cb2934c85a67fc5720afed24cf196a171fdfe4c459061f Nov 02 19:59:44.118534 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:44.118504 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-586f56887b-fzjpt"] Nov 02 19:59:44.119094 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:44.119057 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" podStartSLOduration=2.350477394 podStartE2EDuration="3.11904534s" totalImagesPullingTime="768.567946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:44.11774489 +0000 UTC m=+420.925529117" watchObservedRunningTime="2026-11-02 19:59:44.11904534 +0000 UTC m=+420.926829556" Nov 02 19:59:44.968410 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:44.968369 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586f56887b-fzjpt" event={"ID":"e32544f6-220d-47a8-8155-518c53a314fe","Type":"ContainerStarted","Data":"622efcbefb679a0554cb2934c85a67fc5720afed24cf196a171fdfe4c459061f"} Nov 02 19:59:48.988424 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:48.988381 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586f56887b-fzjpt" event={"ID":"e32544f6-220d-47a8-8155-518c53a314fe","Type":"ContainerStarted","Data":"ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48"} Nov 02 19:59:49.098897 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:49.098818 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-586f56887b-fzjpt" podStartSLOduration=2.105122079 podStartE2EDuration="6.098804583s" totalImagesPullingTime="3.993682504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:49.098239485 +0000 UTC m=+425.906023701" watchObservedRunningTime="2026-11-02 19:59:49.098804583 +0000 UTC m=+425.906588850" Nov 02 19:59:49.974026 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:49.973993 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 19:59:50.131121 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.131085 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r"] Nov 02 19:59:50.131569 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.131338 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" podUID="def7ada1-4544-4fab-800e-8375b1d1983a" containerName="maas-api" containerID="cri-o://0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5" gracePeriod=30 Nov 02 19:59:50.373828 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.373802 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:50.430616 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.430589 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def7ada1-4544-4fab-800e-8375b1d1983a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def7ada1-4544-4fab-800e-8375b1d1983a-openshift-service-ca\") pod \"def7ada1-4544-4fab-800e-8375b1d1983a\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " Nov 02 19:59:50.430726 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.430633 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def7ada1-4544-4fab-800e-8375b1d1983a-kube-api-access-r4748\" (UniqueName: \"kubernetes.io/projected/def7ada1-4544-4fab-800e-8375b1d1983a-kube-api-access-r4748\") pod \"def7ada1-4544-4fab-800e-8375b1d1983a\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " Nov 02 19:59:50.430726 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.430667 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-maas-api-tls\") pod \"def7ada1-4544-4fab-800e-8375b1d1983a\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " Nov 02 19:59:50.430726 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.430687 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-metrics-certs\") pod \"def7ada1-4544-4fab-800e-8375b1d1983a\" (UID: \"def7ada1-4544-4fab-800e-8375b1d1983a\") " Nov 02 19:59:50.431004 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.430980 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def7ada1-4544-4fab-800e-8375b1d1983a-openshift-service-ca" pod "def7ada1-4544-4fab-800e-8375b1d1983a" (UID: "def7ada1-4544-4fab-800e-8375b1d1983a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:59:50.432859 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.432834 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-maas-api-tls" pod "def7ada1-4544-4fab-800e-8375b1d1983a" (UID: "def7ada1-4544-4fab-800e-8375b1d1983a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:59:50.432976 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.432854 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-metrics-certs" pod "def7ada1-4544-4fab-800e-8375b1d1983a" (UID: "def7ada1-4544-4fab-800e-8375b1d1983a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:59:50.432976 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.432911 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def7ada1-4544-4fab-800e-8375b1d1983a-kube-api-access-r4748" pod "def7ada1-4544-4fab-800e-8375b1d1983a" (UID: "def7ada1-4544-4fab-800e-8375b1d1983a"). InnerVolumeSpecName "kube-api-access-r4748". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:59:50.531351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.531271 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/def7ada1-4544-4fab-800e-8375b1d1983a-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:50.531351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.531303 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4748\" (UniqueName: \"kubernetes.io/projected/def7ada1-4544-4fab-800e-8375b1d1983a-kube-api-access-r4748\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:50.531351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.531315 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:50.531351 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.531323 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/def7ada1-4544-4fab-800e-8375b1d1983a-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:50.997696 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.997662 5561 generic.go:356] "Generic (PLEG): container finished" podID="def7ada1-4544-4fab-800e-8375b1d1983a" containerID="0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5" exitCode=0 Nov 02 19:59:50.997939 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.997719 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" Nov 02 19:59:50.997939 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.997714 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" event={"ID":"def7ada1-4544-4fab-800e-8375b1d1983a","Type":"ContainerDied","Data":"0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5"} Nov 02 19:59:50.997939 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.997834 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r" event={"ID":"def7ada1-4544-4fab-800e-8375b1d1983a","Type":"ContainerDied","Data":"2c31846090cb6027cfdc9985dad7cd39873d6230b8410f75e0841a1f418ce851"} Nov 02 19:59:50.997939 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:50.997854 5561 scope.go:122] "RemoveContainer" containerID="0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5" Nov 02 19:59:51.006740 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:51.006720 5561 scope.go:122] "RemoveContainer" containerID="0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5" Nov 02 19:59:51.007009 ip-10-0-134-194 kubenswrapper[5561]: E1102 19:59:51.006992 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5\": container with ID starting with 0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5 not found: ID does not exist" containerID="0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5" Nov 02 19:59:51.007079 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:51.007016 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5"} err="failed to get container status \"0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5\": rpc error: code = NotFound desc = could not find container \"0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5\": container with ID starting with 0bc44a0ea7d369b97b49c186cb0ec54e0b46b587cdbc4e642c4b0e08e8b609a5 not found: ID does not exist" Nov 02 19:59:51.151805 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:51.151770 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r"] Nov 02 19:59:51.199174 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:51.199143 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-t9h7r"] Nov 02 19:59:51.310847 ip-10-0-134-194 kubenswrapper[5561]: I1102 19:59:51.310763 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def7ada1-4544-4fab-800e-8375b1d1983a" path="/var/lib/kubelet/pods/def7ada1-4544-4fab-800e-8375b1d1983a/volumes" Nov 02 20:00:03.778308 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:00:03.778266 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:00:33.778289 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:00:33.778249 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:00:43.448220 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:43.448184 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:00:50.579412 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.579371 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-949s5"] Nov 02 20:00:50.580060 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.579905 5561 cpu_manager.go:414] "containerMap: removing container" podUID="def7ada1-4544-4fab-800e-8375b1d1983a" containerName="maas-api" Nov 02 20:00:50.580060 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.579926 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def7ada1-4544-4fab-800e-8375b1d1983a" containerName="maas-api" Nov 02 20:00:50.580060 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.580030 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="def7ada1-4544-4fab-800e-8375b1d1983a" containerName="maas-api" Nov 02 20:00:50.584270 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.584246 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" Nov 02 20:00:50.635932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.635900 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-949s5"] Nov 02 20:00:50.682350 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.682314 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 20:00:50.682506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.682397 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nddwp\"" Nov 02 20:00:50.720489 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.720453 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frzws\" (UniqueName: \"kubernetes.io/projected/5f1822ae-a4be-4b62-879a-55dd8c652f5c-kube-api-access-frzws\") pod \"llmisvc-controller-manager-6466c554d8-949s5\" (UID: \"5f1822ae-a4be-4b62-879a-55dd8c652f5c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" Nov 02 20:00:50.720617 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.720519 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f1822ae-a4be-4b62-879a-55dd8c652f5c-cert\") pod \"llmisvc-controller-manager-6466c554d8-949s5\" (UID: \"5f1822ae-a4be-4b62-879a-55dd8c652f5c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" Nov 02 20:00:50.821308 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.821275 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f1822ae-a4be-4b62-879a-55dd8c652f5c-cert\") pod \"llmisvc-controller-manager-6466c554d8-949s5\" (UID: \"5f1822ae-a4be-4b62-879a-55dd8c652f5c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" Nov 02 20:00:50.821516 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.821366 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frzws\" (UniqueName: \"kubernetes.io/projected/5f1822ae-a4be-4b62-879a-55dd8c652f5c-kube-api-access-frzws\") pod \"llmisvc-controller-manager-6466c554d8-949s5\" (UID: \"5f1822ae-a4be-4b62-879a-55dd8c652f5c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" Nov 02 20:00:50.823804 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.823778 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f1822ae-a4be-4b62-879a-55dd8c652f5c-cert\") pod \"llmisvc-controller-manager-6466c554d8-949s5\" (UID: \"5f1822ae-a4be-4b62-879a-55dd8c652f5c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" Nov 02 20:00:50.876158 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.876085 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frzws\" (UniqueName: \"kubernetes.io/projected/5f1822ae-a4be-4b62-879a-55dd8c652f5c-kube-api-access-frzws\") pod \"llmisvc-controller-manager-6466c554d8-949s5\" (UID: \"5f1822ae-a4be-4b62-879a-55dd8c652f5c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" Nov 02 20:00:50.896733 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:50.896702 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" Nov 02 20:00:51.262762 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:51.262722 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-949s5"] Nov 02 20:00:51.276498 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:00:51.276463 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1822ae_a4be_4b62_879a_55dd8c652f5c.slice/crio-ddf6f149a3a2909e535de46952a2e2a8c0ec433bd90f82c63b567acb031d9085 WatchSource:0}: Error finding container ddf6f149a3a2909e535de46952a2e2a8c0ec433bd90f82c63b567acb031d9085: Status 404 returned error can't find the container with id ddf6f149a3a2909e535de46952a2e2a8c0ec433bd90f82c63b567acb031d9085 Nov 02 20:00:52.247260 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:52.247216 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" event={"ID":"5f1822ae-a4be-4b62-879a-55dd8c652f5c","Type":"ContainerStarted","Data":"ddf6f149a3a2909e535de46952a2e2a8c0ec433bd90f82c63b567acb031d9085"} Nov 02 20:00:54.257252 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:54.257213 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" event={"ID":"5f1822ae-a4be-4b62-879a-55dd8c652f5c","Type":"ContainerStarted","Data":"98c04f5f45a913574373a8b525a9486c8ae375d6b6156f7885f123d5d334abe1"} Nov 02 20:00:54.257643 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:54.257352 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" Nov 02 20:00:54.364685 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:00:54.364630 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" podStartSLOduration=2.180723346 podStartE2EDuration="4.364616103s" totalImagesPullingTime="2.183892757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:54.36399956 +0000 UTC m=+491.171783801" watchObservedRunningTime="2026-11-02 20:00:54.364616103 +0000 UTC m=+491.172400357" Nov 02 20:01:03.778614 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:01:03.778569 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:01:10.504139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:01:10.504077 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:17.083203 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:01:17.083168 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:25.263845 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:01:25.263811 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-949s5" Nov 02 20:01:26.219483 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:01:26.219451 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:26.398377 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:01:26.398319 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:30.905703 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:01:30.905654 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:33.777954 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:01:33.777917 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:01:43.766449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:01:43.766415 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:50.801454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:01:50.801419 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:01:58.118067 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:01:58.117980 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:01.171349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:01.171315 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:03.779626 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:02:03.779588 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:02:04.597890 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:04.597850 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:08.053937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:08.053898 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:12.236464 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:12.236428 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:12.966198 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:12.966165 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:13.148080 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:13.148043 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:16.122741 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:16.122696 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:19.523007 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.522967 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x"] Nov 02 20:02:19.526232 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.526215 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.626402 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.626361 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 20:02:19.626402 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.626361 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 20:02:19.626650 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.626405 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d2qwk\"" Nov 02 20:02:19.626650 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.626438 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 20:02:19.627389 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.627358 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x"] Nov 02 20:02:19.664074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.663969 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.664074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.664014 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.664354 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.664146 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9tzf\" (UniqueName: \"kubernetes.io/projected/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-kube-api-access-l9tzf\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.664354 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.664190 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.664354 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.664292 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.664354 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.664337 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.668064 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.668043 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:19.765139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.765087 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.765139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.765149 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.765399 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.765168 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.765399 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.765206 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9tzf\" (UniqueName: \"kubernetes.io/projected/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-kube-api-access-l9tzf\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.765399 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.765228 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.765399 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.765260 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.765593 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.765562 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.765648 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.765587 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.765687 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.765653 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.767359 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.767339 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.767684 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.767665 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.819195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.819115 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9tzf\" (UniqueName: \"kubernetes.io/projected/e8b4ee4a-ebb3-4872-bb79-520fec0376ac-kube-api-access-l9tzf\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x\" (UID: \"e8b4ee4a-ebb3-4872-bb79-520fec0376ac\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:19.838063 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:19.838018 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:20.204010 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:20.203976 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x"] Nov 02 20:02:20.212807 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:02:20.212778 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8b4ee4a_ebb3_4872_bb79_520fec0376ac.slice/crio-a07d5f89ffe7e4e78fea7818c5d9d47fa2687c2637c72f70d500860d3fbcc745 WatchSource:0}: Error finding container a07d5f89ffe7e4e78fea7818c5d9d47fa2687c2637c72f70d500860d3fbcc745: Status 404 returned error can't find the container with id a07d5f89ffe7e4e78fea7818c5d9d47fa2687c2637c72f70d500860d3fbcc745 Nov 02 20:02:20.214140 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:20.214122 5561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:02:20.607494 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:20.607413 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" event={"ID":"e8b4ee4a-ebb3-4872-bb79-520fec0376ac","Type":"ContainerStarted","Data":"a07d5f89ffe7e4e78fea7818c5d9d47fa2687c2637c72f70d500860d3fbcc745"} Nov 02 20:02:22.617713 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:22.617678 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" event={"ID":"e8b4ee4a-ebb3-4872-bb79-520fec0376ac","Type":"ContainerStarted","Data":"50f88b99b763871ccd4ed393d23ef6208ffc3860289c08628faac553a5d24371"} Nov 02 20:02:22.727318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:22.727266 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podStartSLOduration=2.15874651 podStartE2EDuration="3.727254241s" totalImagesPullingTime="1.568507731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:22.72677478 +0000 UTC m=+579.534559012" watchObservedRunningTime="2026-11-02 20:02:22.727254241 +0000 UTC m=+579.535038456" Nov 02 20:02:22.838576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:22.838540 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:29.838966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:29.838929 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:29.851497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:29.851473 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:30.663599 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:30.663567 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:02:31.188813 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:31.188776 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:33.777647 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:02:33.777611 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:02:39.949581 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:39.949518 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:40.123425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:40.123392 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:43.775102 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.775069 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s"] Nov 02 20:02:43.778377 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.778360 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.828309 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.828285 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 20:02:43.858040 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.858012 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fdd8j\"" Nov 02 20:02:43.870826 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.870801 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.870958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.870847 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.870958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.870944 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b49x4\" (UniqueName: \"kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-kube-api-access-b49x4\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.871084 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.871057 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.871141 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.871099 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e64aba5-a055-4e97-8a83-dc814e961e89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.871196 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.871141 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.871196 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.871173 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.871196 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.871189 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.871343 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.871220 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.876355 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.876326 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s"] Nov 02 20:02:43.972059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972032 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972216 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972076 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972216 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972105 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b49x4\" (UniqueName: \"kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-kube-api-access-b49x4\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972216 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972148 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972216 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972173 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e64aba5-a055-4e97-8a83-dc814e961e89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972216 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972205 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972370 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972413 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972459 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972462 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972710 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972517 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972785 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972719 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972785 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972730 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.972993 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.972971 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e64aba5-a055-4e97-8a83-dc814e961e89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.974384 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.974360 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:43.974816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:43.974797 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:44.025855 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:44.025790 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:44.025855 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:44.025812 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b49x4\" (UniqueName: \"kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-kube-api-access-b49x4\") pod \"maas-default-gateway-openshift-default-696f47c6c-qk82s\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:44.086567 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:44.086535 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:44.089442 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:44.089421 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:44.366799 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:44.366723 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:44.507718 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:44.507680 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s"] Nov 02 20:02:44.515768 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:02:44.515737 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e64aba5_a055_4e97_8a83_dc814e961e89.slice/crio-14e60311c775bac0ef0a56abf0a314457a189ef0ec220805b1d71b509efd56c7 WatchSource:0}: Error finding container 14e60311c775bac0ef0a56abf0a314457a189ef0ec220805b1d71b509efd56c7: Status 404 returned error can't find the container with id 14e60311c775bac0ef0a56abf0a314457a189ef0ec220805b1d71b509efd56c7 Nov 02 20:02:44.712437 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:44.712401 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" event={"ID":"8e64aba5-a055-4e97-8a83-dc814e961e89","Type":"ContainerStarted","Data":"14e60311c775bac0ef0a56abf0a314457a189ef0ec220805b1d71b509efd56c7"} Nov 02 20:02:45.076385 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:45.076310 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cdd9f4f95-95knq"] Nov 02 20:02:45.080969 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:45.080943 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cdd9f4f95-95knq" Nov 02 20:02:45.130661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:45.130628 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cdd9f4f95-95knq"] Nov 02 20:02:45.183433 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:45.183398 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82jkg\" (UniqueName: \"kubernetes.io/projected/b9ed36fe-18cd-49a5-819b-a60d0f7fadaa-kube-api-access-82jkg\") pod \"authorino-5cdd9f4f95-95knq\" (UID: \"b9ed36fe-18cd-49a5-819b-a60d0f7fadaa\") " pod="kuadrant-system/authorino-5cdd9f4f95-95knq" Nov 02 20:02:45.283824 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:45.283790 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:45.284076 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:45.283894 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82jkg\" (UniqueName: \"kubernetes.io/projected/b9ed36fe-18cd-49a5-819b-a60d0f7fadaa-kube-api-access-82jkg\") pod \"authorino-5cdd9f4f95-95knq\" (UID: \"b9ed36fe-18cd-49a5-819b-a60d0f7fadaa\") " pod="kuadrant-system/authorino-5cdd9f4f95-95knq" Nov 02 20:02:45.338082 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:45.338008 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82jkg\" (UniqueName: \"kubernetes.io/projected/b9ed36fe-18cd-49a5-819b-a60d0f7fadaa-kube-api-access-82jkg\") pod \"authorino-5cdd9f4f95-95knq\" (UID: \"b9ed36fe-18cd-49a5-819b-a60d0f7fadaa\") " pod="kuadrant-system/authorino-5cdd9f4f95-95knq" Nov 02 20:02:45.393826 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:45.393790 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cdd9f4f95-95knq" Nov 02 20:02:45.535830 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:45.535778 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:45.763767 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:45.763733 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cdd9f4f95-95knq"] Nov 02 20:02:45.772221 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:02:45.772189 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9ed36fe_18cd_49a5_819b_a60d0f7fadaa.slice/crio-20963ed11ffe3e8513387398f42190733b243145d00a6c1de71edd838aa4f1da WatchSource:0}: Error finding container 20963ed11ffe3e8513387398f42190733b243145d00a6c1de71edd838aa4f1da: Status 404 returned error can't find the container with id 20963ed11ffe3e8513387398f42190733b243145d00a6c1de71edd838aa4f1da Nov 02 20:02:46.724087 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:46.724042 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cdd9f4f95-95knq" event={"ID":"b9ed36fe-18cd-49a5-819b-a60d0f7fadaa","Type":"ContainerStarted","Data":"20963ed11ffe3e8513387398f42190733b243145d00a6c1de71edd838aa4f1da"} Nov 02 20:02:47.730627 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:47.730544 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cdd9f4f95-95knq" event={"ID":"b9ed36fe-18cd-49a5-819b-a60d0f7fadaa","Type":"ContainerStarted","Data":"0137c02ed2cfef8064b6fbf4c215557cacdabc82deae969b8fd0a376c70aa2f6"} Nov 02 20:02:47.836286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:47.836222 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5cdd9f4f95-95knq" podStartSLOduration=1.159513396 podStartE2EDuration="2.83620287s" totalImagesPullingTime="1.676689474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:47.836087349 +0000 UTC m=+604.643871568" watchObservedRunningTime="2026-11-02 20:02:47.83620287 +0000 UTC m=+604.643987087" Nov 02 20:02:47.885047 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:47.885010 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-586f56887b-fzjpt"] Nov 02 20:02:47.885275 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:47.885246 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-586f56887b-fzjpt" podUID="e32544f6-220d-47a8-8155-518c53a314fe" containerName="authorino" containerID="cri-o://ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48" gracePeriod=30 Nov 02 20:02:48.154670 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.154645 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586f56887b-fzjpt" Nov 02 20:02:48.213396 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.213367 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32544f6-220d-47a8-8155-518c53a314fe-kube-api-access-h2csn\" (UniqueName: \"kubernetes.io/projected/e32544f6-220d-47a8-8155-518c53a314fe-kube-api-access-h2csn\") pod \"e32544f6-220d-47a8-8155-518c53a314fe\" (UID: \"e32544f6-220d-47a8-8155-518c53a314fe\") " Nov 02 20:02:48.215868 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.215841 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32544f6-220d-47a8-8155-518c53a314fe-kube-api-access-h2csn" pod "e32544f6-220d-47a8-8155-518c53a314fe" (UID: "e32544f6-220d-47a8-8155-518c53a314fe"). InnerVolumeSpecName "kube-api-access-h2csn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:02:48.314984 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.314863 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2csn\" (UniqueName: \"kubernetes.io/projected/e32544f6-220d-47a8-8155-518c53a314fe-kube-api-access-h2csn\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:02:48.638570 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.638539 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:02:48.638653 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.638606 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:02:48.638653 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.638639 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:02:48.736437 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.736404 5561 generic.go:356] "Generic (PLEG): container finished" podID="e32544f6-220d-47a8-8155-518c53a314fe" containerID="ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48" exitCode=0 Nov 02 20:02:48.736963 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.736462 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586f56887b-fzjpt" Nov 02 20:02:48.736963 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.736513 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586f56887b-fzjpt" event={"ID":"e32544f6-220d-47a8-8155-518c53a314fe","Type":"ContainerDied","Data":"ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48"} Nov 02 20:02:48.736963 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.736545 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586f56887b-fzjpt" event={"ID":"e32544f6-220d-47a8-8155-518c53a314fe","Type":"ContainerDied","Data":"622efcbefb679a0554cb2934c85a67fc5720afed24cf196a171fdfe4c459061f"} Nov 02 20:02:48.736963 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.736562 5561 scope.go:122] "RemoveContainer" containerID="ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48" Nov 02 20:02:48.743966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.743380 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" event={"ID":"8e64aba5-a055-4e97-8a83-dc814e961e89","Type":"ContainerStarted","Data":"b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b"} Nov 02 20:02:48.754203 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.753695 5561 scope.go:122] "RemoveContainer" containerID="ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48" Nov 02 20:02:48.754810 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:02:48.754772 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48\": container with ID starting with ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48 not found: ID does not exist" containerID="ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48" Nov 02 20:02:48.754938 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.754814 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48"} err="failed to get container status \"ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48\": rpc error: code = NotFound desc = could not find container \"ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48\": container with ID starting with ae6c777f755104f8fab0cc696630670ace4eaf7fefac5c487c1dcb74c1725a48 not found: ID does not exist" Nov 02 20:02:48.890093 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.890006 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-586f56887b-fzjpt"] Nov 02 20:02:48.896050 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:48.896017 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-586f56887b-fzjpt"] Nov 02 20:02:49.005401 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:49.005339 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" podStartSLOduration=1.885346906 podStartE2EDuration="6.005322269s" totalImagesPullingTime="4.119975363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:49.00433889 +0000 UTC m=+605.812123133" watchObservedRunningTime="2026-11-02 20:02:49.005322269 +0000 UTC m=+605.813106485" Nov 02 20:02:49.090527 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:49.090479 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:49.091982 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:49.091946 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qk82s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 20:02:49.092132 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:49.092018 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 20:02:49.312886 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:49.312842 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32544f6-220d-47a8-8155-518c53a314fe" path="/var/lib/kubelet/pods/e32544f6-220d-47a8-8155-518c53a314fe/volumes" Nov 02 20:02:50.090497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:50.090456 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qk82s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 20:02:50.090912 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:50.090522 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 20:02:51.090122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:51.090076 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qk82s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 20:02:51.090282 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:51.090153 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 20:02:52.090828 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:52.090790 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qk82s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 20:02:52.091240 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:52.090860 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 20:02:53.089972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:53.089940 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qk82s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 20:02:53.090172 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:53.089999 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 20:02:54.090570 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:54.090535 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qk82s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 20:02:54.091083 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:54.090600 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 20:02:55.090577 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:55.090541 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qk82s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 20:02:55.091124 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:55.090618 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 20:02:56.090167 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:56.090123 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qk82s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 02 20:02:56.090336 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:56.090193 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 02 20:02:56.453964 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:56.453930 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:56.632399 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:56.632348 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:02:58.131053 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:58.131008 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qk82s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:02:58.131574 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:58.131099 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:02:58.870489 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:58.870460 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:58.870788 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:58.870765 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:02:58.871421 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:02:58.871401 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:03:03.778145 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:03:03.778104 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:03:19.193492 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:19.193448 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:29.666779 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:29.666745 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:31.262570 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:31.262528 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:31.464660 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:31.464625 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:32.813674 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:32.813645 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:33.027040 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:33.027008 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:33.779869 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:03:33.779824 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:03:44.407092 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:44.407051 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:56.785678 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:56.785643 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:56.966096 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:56.966057 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:57.395379 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:57.395345 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:57.598750 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:57.598716 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:57.829866 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:57.829836 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:58.012770 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:58.012718 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:58.216405 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:58.216372 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:58.422178 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:58.422142 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:58.598385 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:58.598302 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:58.770468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:58.770428 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:58.945571 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:58.945538 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:59.084055 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.084023 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9"] Nov 02 20:03:59.084363 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.084350 5561 cpu_manager.go:414] "containerMap: removing container" podUID="e32544f6-220d-47a8-8155-518c53a314fe" containerName="authorino" Nov 02 20:03:59.084410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.084363 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32544f6-220d-47a8-8155-518c53a314fe" containerName="authorino" Nov 02 20:03:59.084444 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.084440 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32544f6-220d-47a8-8155-518c53a314fe" containerName="authorino" Nov 02 20:03:59.086343 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.086308 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.135967 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.135936 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2052e3-openshift-default-dockercfg-8r7gl\"" Nov 02 20:03:59.144988 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.144960 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.145216 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.145189 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.145304 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.145271 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.145365 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.145311 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.145421 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.145386 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg974\" (UniqueName: \"kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-kube-api-access-pg974\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.145487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.145474 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.145545 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.145509 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.145599 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.145557 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.145650 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.145593 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386e14bc-708a-4130-9b7e-976e11cc5e24-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.145734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.145700 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:59.184327 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.184292 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9"] Nov 02 20:03:59.246305 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.246186 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.246305 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.246265 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.246514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.246336 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.246514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.246370 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386e14bc-708a-4130-9b7e-976e11cc5e24-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.246514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.246427 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.246514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.246454 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.246514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.246479 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.246514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.246504 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.246744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.246527 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg974\" (UniqueName: \"kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-kube-api-access-pg974\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.247024 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.246994 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.247253 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.247022 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.247394 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.247369 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.247560 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.247303 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.248018 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.247973 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386e14bc-708a-4130-9b7e-976e11cc5e24-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.249247 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.249220 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.249344 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.249323 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.300840 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.300802 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.301037 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.300851 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg974\" (UniqueName: \"kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-kube-api-access-pg974\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.318357 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.318326 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:59.397757 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.397729 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:03:59.544329 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.544199 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:03:59.824940 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.824849 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9"] Nov 02 20:03:59.832730 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:03:59.832692 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386e14bc_708a_4130_9b7e_976e11cc5e24.slice/crio-4a6ee2c4afb2ea2c6cc0f79bb011f051b252deff2fe21e1ac1daeb8ad2300acb WatchSource:0}: Error finding container 4a6ee2c4afb2ea2c6cc0f79bb011f051b252deff2fe21e1ac1daeb8ad2300acb: Status 404 returned error can't find the container with id 4a6ee2c4afb2ea2c6cc0f79bb011f051b252deff2fe21e1ac1daeb8ad2300acb Nov 02 20:03:59.835062 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.835014 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:03:59.835172 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.835136 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:03:59.835230 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:03:59.835183 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:00.051157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:00.051126 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" event={"ID":"386e14bc-708a-4130-9b7e-976e11cc5e24","Type":"ContainerStarted","Data":"9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0"} Nov 02 20:04:00.051541 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:00.051177 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" event={"ID":"386e14bc-708a-4130-9b7e-976e11cc5e24","Type":"ContainerStarted","Data":"4a6ee2c4afb2ea2c6cc0f79bb011f051b252deff2fe21e1ac1daeb8ad2300acb"} Nov 02 20:04:00.163175 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:00.163120 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" podStartSLOduration=1.163106624 podStartE2EDuration="1.163106624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:00.161973569 +0000 UTC m=+676.969757811" watchObservedRunningTime="2026-11-02 20:04:00.163106624 +0000 UTC m=+676.970890830" Nov 02 20:04:00.398148 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:00.398111 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:04:00.403202 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:00.403176 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:04:01.056756 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:01.056726 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:04:01.057804 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:01.057778 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:04:02.086893 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:02.086855 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:02.273286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:02.273254 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:02.454599 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:02.454565 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:02.693662 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:02.693606 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:02.901905 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:02.901802 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:03.087664 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:03.087621 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:03.370845 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:03.370811 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:03.559818 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:03.559782 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:03.778643 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:04:03.778601 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:04:03.966679 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:03.966647 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:04.174941 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:04.174910 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:04.533071 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:04.532992 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:04.725557 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:04.725520 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:04.901482 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:04.901403 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:06.764322 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:06.764287 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:06.944380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:06.944345 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:07.886134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:07.886102 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh"] Nov 02 20:04:07.888568 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:07.888548 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:07.942269 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:07.942233 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh"] Nov 02 20:04:07.988276 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:07.988241 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-cc64d3-serving-cert\"" Nov 02 20:04:08.022972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.022932 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx7qd\" (UniqueName: \"kubernetes.io/projected/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-kube-api-access-xx7qd\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.022972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.022988 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-metrics-certs\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.023201 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.023042 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.023201 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.023081 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.115195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.115157 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:08.124245 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.124217 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx7qd\" (UniqueName: \"kubernetes.io/projected/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-kube-api-access-xx7qd\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.124397 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.124264 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-metrics-certs\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.124397 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.124285 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.124397 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.124335 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.124528 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:04:08.124456 5561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-serving-cert: secret "maas-api-e2e-worker-w4-cc64d3-serving-cert" not found Nov 02 20:04:08.124567 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:04:08.124533 5561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-maas-api-tls podName:1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d nodeName:}" failed. No retries permitted until 2026-11-02 20:04:08.624513044 +0000 UTC m=+685.432297243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-maas-api-tls") pod "maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" (UID: "1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d") : secret "maas-api-e2e-worker-w4-cc64d3-serving-cert" not found Nov 02 20:04:08.125156 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.125135 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.126990 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.126968 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-metrics-certs\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.178692 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.178666 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx7qd\" (UniqueName: \"kubernetes.io/projected/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-kube-api-access-xx7qd\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.285692 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.285641 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:08.453141 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.453042 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:08.628723 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.628633 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.631286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.631259 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:08.800707 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:08.800620 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:09.166337 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:09.166303 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh"] Nov 02 20:04:09.174396 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:04:09.174364 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f198ce3_2bfb_4b2d_bda5_cd81ff90e37d.slice/crio-fc95320918028c8af32cba6ca8091e97455af7c39dcfccaf551b18b12b13eb5c WatchSource:0}: Error finding container fc95320918028c8af32cba6ca8091e97455af7c39dcfccaf551b18b12b13eb5c: Status 404 returned error can't find the container with id fc95320918028c8af32cba6ca8091e97455af7c39dcfccaf551b18b12b13eb5c Nov 02 20:04:10.097372 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:10.097343 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" event={"ID":"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d","Type":"ContainerStarted","Data":"fc95320918028c8af32cba6ca8091e97455af7c39dcfccaf551b18b12b13eb5c"} Nov 02 20:04:11.102538 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:11.102493 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" event={"ID":"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d","Type":"ContainerStarted","Data":"3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351"} Nov 02 20:04:11.102999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:11.102721 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:11.211151 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:11.211090 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" podStartSLOduration=3.319595893 podStartE2EDuration="4.211069962s" totalImagesPullingTime="891.474069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:11.210356468 +0000 UTC m=+688.018140703" watchObservedRunningTime="2026-11-02 20:04:11.211069962 +0000 UTC m=+688.018854175" Nov 02 20:04:11.690933 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:11.690892 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:11.885340 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:11.885302 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:12.077259 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.077169 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:12.270926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.270898 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:12.791812 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.791780 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr"] Nov 02 20:04:12.794229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.794208 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:12.844094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.844056 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-982b8a-serving-cert\"" Nov 02 20:04:12.867775 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.867714 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/429ad3de-f074-44d6-83ac-b8cf9ce545cd-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:12.867997 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.867786 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-metrics-certs\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:12.867997 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.867827 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbvfd\" (UniqueName: \"kubernetes.io/projected/429ad3de-f074-44d6-83ac-b8cf9ce545cd-kube-api-access-gbvfd\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:12.867997 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.867981 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-maas-api-tls\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:12.892646 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.892617 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr"] Nov 02 20:04:12.969363 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.969325 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/429ad3de-f074-44d6-83ac-b8cf9ce545cd-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:12.969538 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.969374 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-metrics-certs\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:12.969538 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.969493 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbvfd\" (UniqueName: \"kubernetes.io/projected/429ad3de-f074-44d6-83ac-b8cf9ce545cd-kube-api-access-gbvfd\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:12.969613 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.969552 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-maas-api-tls\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:12.969704 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:04:12.969686 5561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-serving-cert: secret "maas-api-e2e-worker-w6-982b8a-serving-cert" not found Nov 02 20:04:12.969761 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:04:12.969751 5561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-maas-api-tls podName:429ad3de-f074-44d6-83ac-b8cf9ce545cd nodeName:}" failed. No retries permitted until 2026-11-02 20:04:13.469731307 +0000 UTC m=+690.277515510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-maas-api-tls") pod "maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" (UID: "429ad3de-f074-44d6-83ac-b8cf9ce545cd") : secret "maas-api-e2e-worker-w6-982b8a-serving-cert" not found Nov 02 20:04:12.970055 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.970023 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/429ad3de-f074-44d6-83ac-b8cf9ce545cd-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:12.971925 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:12.971905 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-metrics-certs\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:13.025815 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:13.025775 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbvfd\" (UniqueName: \"kubernetes.io/projected/429ad3de-f074-44d6-83ac-b8cf9ce545cd-kube-api-access-gbvfd\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:13.033264 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:13.033233 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:13.204544 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:13.204507 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:13.408965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:13.408931 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:13.474132 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:13.474037 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-maas-api-tls\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:13.476553 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:13.476516 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-maas-api-tls\") pod \"maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:13.701709 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:13.701678 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:13.706693 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:13.706668 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:14.074285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:14.074200 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr"] Nov 02 20:04:14.082439 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:04:14.082412 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429ad3de_f074_44d6_83ac_b8cf9ce545cd.slice/crio-406cd3bd9fa09426611002d0e89a2a352ff00263799ef2b9f2fa037ec935b014 WatchSource:0}: Error finding container 406cd3bd9fa09426611002d0e89a2a352ff00263799ef2b9f2fa037ec935b014: Status 404 returned error can't find the container with id 406cd3bd9fa09426611002d0e89a2a352ff00263799ef2b9f2fa037ec935b014 Nov 02 20:04:14.117288 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:14.117252 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" event={"ID":"429ad3de-f074-44d6-83ac-b8cf9ce545cd","Type":"ContainerStarted","Data":"406cd3bd9fa09426611002d0e89a2a352ff00263799ef2b9f2fa037ec935b014"} Nov 02 20:04:14.338939 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:14.338838 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:14.514687 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:14.514640 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:15.124809 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:15.124535 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" event={"ID":"429ad3de-f074-44d6-83ac-b8cf9ce545cd","Type":"ContainerStarted","Data":"800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe"} Nov 02 20:04:15.124809 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:15.124703 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:15.247609 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:15.247558 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" podStartSLOduration=2.296176824 podStartE2EDuration="3.247543564s" totalImagesPullingTime="951.36674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:15.247129499 +0000 UTC m=+692.054913739" watchObservedRunningTime="2026-11-02 20:04:15.247543564 +0000 UTC m=+692.055327780" Nov 02 20:04:16.891758 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:16.891722 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:17.068457 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:17.068411 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:17.112366 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:17.112335 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:04:17.785482 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:17.785443 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh"] Nov 02 20:04:17.788951 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:17.788933 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:17.838561 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:17.838530 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-ae8e6e-serving-cert\"" Nov 02 20:04:17.887400 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:17.887362 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh"] Nov 02 20:04:17.915978 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:17.915940 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8qtp\" (UniqueName: \"kubernetes.io/projected/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-kube-api-access-k8qtp\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:17.915978 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:17.915982 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:17.916522 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:17.916012 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-metrics-certs\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:17.916522 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:17.916156 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:18.015349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.015309 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:18.017424 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.017399 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8qtp\" (UniqueName: \"kubernetes.io/projected/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-kube-api-access-k8qtp\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:18.017494 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.017434 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:18.017494 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.017460 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-metrics-certs\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:18.017597 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.017529 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:18.017674 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:04:18.017658 5561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-serving-cert: secret "maas-api-e2e-worker-w5-ae8e6e-serving-cert" not found Nov 02 20:04:18.017767 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:04:18.017755 5561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-maas-api-tls podName:c94eb3e8-cfd0-46ca-8b24-6746f4437e4e nodeName:}" failed. No retries permitted until 2026-11-02 20:04:18.517733927 +0000 UTC m=+695.325518125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-maas-api-tls") pod "maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" (UID: "c94eb3e8-cfd0-46ca-8b24-6746f4437e4e") : secret "maas-api-e2e-worker-w5-ae8e6e-serving-cert" not found Nov 02 20:04:18.018283 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.018260 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:18.019941 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.019920 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-metrics-certs\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:18.075699 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.075625 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8qtp\" (UniqueName: \"kubernetes.io/projected/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-kube-api-access-k8qtp\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:18.192133 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.192095 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:18.362149 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.362073 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:18.522995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.522965 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:18.525314 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.525292 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:18.701534 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:18.701495 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:19.068237 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:19.068149 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh"] Nov 02 20:04:19.076285 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:04:19.076244 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94eb3e8_cfd0_46ca_8b24_6746f4437e4e.slice/crio-48cead0bac28b0fc272ff085e7603cfcf03876e6012e45c3a238d23792aa43d8 WatchSource:0}: Error finding container 48cead0bac28b0fc272ff085e7603cfcf03876e6012e45c3a238d23792aa43d8: Status 404 returned error can't find the container with id 48cead0bac28b0fc272ff085e7603cfcf03876e6012e45c3a238d23792aa43d8 Nov 02 20:04:19.144415 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:19.144381 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" event={"ID":"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e","Type":"ContainerStarted","Data":"48cead0bac28b0fc272ff085e7603cfcf03876e6012e45c3a238d23792aa43d8"} Nov 02 20:04:20.150658 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:20.150620 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" event={"ID":"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e","Type":"ContainerStarted","Data":"7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291"} Nov 02 20:04:20.151101 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:20.150750 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:20.269565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:20.269450 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" podStartSLOduration=2.395781939 podStartE2EDuration="3.269435591s" totalImagesPullingTime="873.653652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:20.268329638 +0000 UTC m=+697.076113877" watchObservedRunningTime="2026-11-02 20:04:20.269435591 +0000 UTC m=+697.077219806" Nov 02 20:04:21.134521 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:21.134492 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:04:21.849067 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:21.849029 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:22.025005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:22.024972 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:22.870170 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:22.870139 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7"] Nov 02 20:04:22.873582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:22.873561 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:22.925716 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:22.925686 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-035e47-serving-cert\"" Nov 02 20:04:22.974364 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:22.974332 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7"] Nov 02 20:04:23.069032 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.068997 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05959640-ae04-4597-b190-232c13f303c2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.069032 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.069042 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.069238 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.069138 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g6fc\" (UniqueName: \"kubernetes.io/projected/05959640-ae04-4597-b190-232c13f303c2-kube-api-access-7g6fc\") pod \"maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.069238 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.069189 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-metrics-certs\") pod \"maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.126777 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.126692 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:23.170105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.170072 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.170296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.170120 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g6fc\" (UniqueName: \"kubernetes.io/projected/05959640-ae04-4597-b190-232c13f303c2-kube-api-access-7g6fc\") pod \"maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.170296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.170238 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-metrics-certs\") pod \"maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.170415 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.170339 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05959640-ae04-4597-b190-232c13f303c2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.170958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.170935 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05959640-ae04-4597-b190-232c13f303c2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.172698 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.172677 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.172804 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.172680 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-metrics-certs\") pod \"maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.224406 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.224375 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g6fc\" (UniqueName: \"kubernetes.io/projected/05959640-ae04-4597-b190-232c13f303c2-kube-api-access-7g6fc\") pod \"maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.320145 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.320111 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:23.485715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.485681 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:23.517323 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.517283 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:23.854565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:23.854484 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7"] Nov 02 20:04:23.862377 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:04:23.862345 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05959640_ae04_4597_b190_232c13f303c2.slice/crio-1d8d1ef69fc5f70f9d2decd5c087ac1a944f489d884f34ddefc9d2c941d560ab WatchSource:0}: Error finding container 1d8d1ef69fc5f70f9d2decd5c087ac1a944f489d884f34ddefc9d2c941d560ab: Status 404 returned error can't find the container with id 1d8d1ef69fc5f70f9d2decd5c087ac1a944f489d884f34ddefc9d2c941d560ab Nov 02 20:04:24.170054 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:24.170014 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" event={"ID":"05959640-ae04-4597-b190-232c13f303c2","Type":"ContainerStarted","Data":"1d8d1ef69fc5f70f9d2decd5c087ac1a944f489d884f34ddefc9d2c941d560ab"} Nov 02 20:04:24.433397 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:24.433308 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:24.619412 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:24.619378 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:25.176816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:25.176780 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" event={"ID":"05959640-ae04-4597-b190-232c13f303c2","Type":"ContainerStarted","Data":"ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819"} Nov 02 20:04:25.177289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:25.176864 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:25.286231 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:25.286181 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" podStartSLOduration=2.5136227079999998 podStartE2EDuration="3.286166095s" totalImagesPullingTime="772.543387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:25.285151965 +0000 UTC m=+702.092936206" watchObservedRunningTime="2026-11-02 20:04:25.286166095 +0000 UTC m=+702.093950311" Nov 02 20:04:26.160818 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:26.160785 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:04:28.222892 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:28.222839 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:28.412002 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:28.411963 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:28.613163 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:28.613084 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:29.329727 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:29.329696 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:29.529796 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:29.529752 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:30.536376 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:30.536333 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:31.185812 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:31.185776 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:04:32.738174 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.738140 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz"] Nov 02 20:04:32.741665 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.741645 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:32.762694 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.762661 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-metrics-certs\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:32.762868 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.762750 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:32.762868 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.762818 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ff587f-ba28-4c80-b51a-11fb3755bad9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:32.763021 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.762922 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4s47\" (UniqueName: \"kubernetes.io/projected/f6ff587f-ba28-4c80-b51a-11fb3755bad9-kube-api-access-n4s47\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:32.840420 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.840378 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-accdf5-serving-cert\"" Nov 02 20:04:32.841762 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.841740 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz"] Nov 02 20:04:32.863617 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.863585 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-metrics-certs\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:32.863714 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.863624 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:32.863714 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.863645 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ff587f-ba28-4c80-b51a-11fb3755bad9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:32.863714 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.863687 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4s47\" (UniqueName: \"kubernetes.io/projected/f6ff587f-ba28-4c80-b51a-11fb3755bad9-kube-api-access-n4s47\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:32.863841 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:04:32.863747 5561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-serving-cert: secret "maas-api-e2e-worker-w1-accdf5-serving-cert" not found Nov 02 20:04:32.863841 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:04:32.863822 5561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-maas-api-tls podName:f6ff587f-ba28-4c80-b51a-11fb3755bad9 nodeName:}" failed. No retries permitted until 2026-11-02 20:04:33.363800286 +0000 UTC m=+710.171584497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-maas-api-tls") pod "maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" (UID: "f6ff587f-ba28-4c80-b51a-11fb3755bad9") : secret "maas-api-e2e-worker-w1-accdf5-serving-cert" not found Nov 02 20:04:32.864419 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.864397 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ff587f-ba28-4c80-b51a-11fb3755bad9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:32.866051 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.866032 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-metrics-certs\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:32.918967 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.918937 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4s47\" (UniqueName: \"kubernetes.io/projected/f6ff587f-ba28-4c80-b51a-11fb3755bad9-kube-api-access-n4s47\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:32.982813 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:32.982765 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:33.159346 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:33.159314 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:33.368917 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:33.368855 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:33.371326 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:33.371303 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:33.557995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:33.557910 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:33.654206 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:33.654166 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:33.779243 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:04:33.779208 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:04:34.024197 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:34.024153 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz"] Nov 02 20:04:34.032455 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:04:34.032424 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ff587f_ba28_4c80_b51a_11fb3755bad9.slice/crio-a81c8ba2e0bc45769a76aed0a6742818f4ff54affaff87fd87b6c9a9ca5caec4 WatchSource:0}: Error finding container a81c8ba2e0bc45769a76aed0a6742818f4ff54affaff87fd87b6c9a9ca5caec4: Status 404 returned error can't find the container with id a81c8ba2e0bc45769a76aed0a6742818f4ff54affaff87fd87b6c9a9ca5caec4 Nov 02 20:04:34.226118 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:34.226087 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" event={"ID":"f6ff587f-ba28-4c80-b51a-11fb3755bad9","Type":"ContainerStarted","Data":"a81c8ba2e0bc45769a76aed0a6742818f4ff54affaff87fd87b6c9a9ca5caec4"} Nov 02 20:04:35.233067 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:35.233031 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" event={"ID":"f6ff587f-ba28-4c80-b51a-11fb3755bad9","Type":"ContainerStarted","Data":"dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e"} Nov 02 20:04:35.233523 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:35.233108 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:35.342552 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:35.342498 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" podStartSLOduration=2.55656764 podStartE2EDuration="3.342483296s" totalImagesPullingTime="785.915656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:35.341198583 +0000 UTC m=+712.148982803" watchObservedRunningTime="2026-11-02 20:04:35.342483296 +0000 UTC m=+712.150267512" Nov 02 20:04:37.779830 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:37.779797 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:41.243632 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:41.243603 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:04:42.079442 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.079398 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj"] Nov 02 20:04:42.092247 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.092216 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.146546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.146509 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj"] Nov 02 20:04:42.148280 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.148251 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-metrics-certs\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.148425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.148300 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.148507 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.148450 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de79c7aa-d3aa-4209-adc4-53d5e54c5968-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.148574 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.148556 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5lq8\" (UniqueName: \"kubernetes.io/projected/de79c7aa-d3aa-4209-adc4-53d5e54c5968-kube-api-access-s5lq8\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.185015 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.184970 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-9fcedd-serving-cert\"" Nov 02 20:04:42.248995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.248960 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.248995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.249008 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de79c7aa-d3aa-4209-adc4-53d5e54c5968-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.249431 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.249058 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lq8\" (UniqueName: \"kubernetes.io/projected/de79c7aa-d3aa-4209-adc4-53d5e54c5968-kube-api-access-s5lq8\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.249431 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.249088 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-metrics-certs\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.249431 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:04:42.249124 5561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-serving-cert: secret "maas-api-e2e-worker-w2-9fcedd-serving-cert" not found Nov 02 20:04:42.249431 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:04:42.249201 5561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-maas-api-tls podName:de79c7aa-d3aa-4209-adc4-53d5e54c5968 nodeName:}" failed. No retries permitted until 2026-11-02 20:04:42.749180828 +0000 UTC m=+719.556965038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-maas-api-tls") pod "maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" (UID: "de79c7aa-d3aa-4209-adc4-53d5e54c5968") : secret "maas-api-e2e-worker-w2-9fcedd-serving-cert" not found Nov 02 20:04:42.249699 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.249680 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de79c7aa-d3aa-4209-adc4-53d5e54c5968-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.251458 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.251441 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-metrics-certs\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.303746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.303718 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5lq8\" (UniqueName: \"kubernetes.io/projected/de79c7aa-d3aa-4209-adc4-53d5e54c5968-kube-api-access-s5lq8\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.325142 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.325101 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:42.493274 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.493236 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:42.756169 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.756076 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:42.756169 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.756153 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:42.758491 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:42.758467 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:43.005626 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:43.005585 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:43.406973 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:43.406937 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj"] Nov 02 20:04:43.413937 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:04:43.413911 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde79c7aa_d3aa_4209_adc4_53d5e54c5968.slice/crio-354a52ab87693881257c45c64b0f7a1aefc9677a5e3905687e41a6541aa40467 WatchSource:0}: Error finding container 354a52ab87693881257c45c64b0f7a1aefc9677a5e3905687e41a6541aa40467: Status 404 returned error can't find the container with id 354a52ab87693881257c45c64b0f7a1aefc9677a5e3905687e41a6541aa40467 Nov 02 20:04:44.273214 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:44.273183 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" event={"ID":"de79c7aa-d3aa-4209-adc4-53d5e54c5968","Type":"ContainerStarted","Data":"354a52ab87693881257c45c64b0f7a1aefc9677a5e3905687e41a6541aa40467"} Nov 02 20:04:44.793842 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:44.793805 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:45.280272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:45.280239 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" event={"ID":"de79c7aa-d3aa-4209-adc4-53d5e54c5968","Type":"ContainerStarted","Data":"aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515"} Nov 02 20:04:45.280454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:45.280390 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:45.413758 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:45.413706 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" podStartSLOduration=2.601929819 podStartE2EDuration="3.41369262s" totalImagesPullingTime="811.762801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:45.41328093 +0000 UTC m=+722.221065165" watchObservedRunningTime="2026-11-02 20:04:45.41369262 +0000 UTC m=+722.221476836" Nov 02 20:04:45.450902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:45.449864 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:47.049084 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:47.049023 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:51.291776 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:51.291701 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:04:51.314011 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:51.313980 5561 generic.go:356] "Generic (PLEG): container finished" podID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerID="9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0" exitCode=0 Nov 02 20:04:51.314169 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:51.314021 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" event={"ID":"386e14bc-708a-4130-9b7e-976e11cc5e24","Type":"ContainerDied","Data":"9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0"} Nov 02 20:04:51.314375 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:51.314362 5561 scope.go:122] "RemoveContainer" containerID="9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0" Nov 02 20:04:51.315582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:51.315554 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:51.315640 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:51.315625 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:51.315680 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:51.315662 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:04:51.561181 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:51.561072 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:52.322074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:52.322036 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" event={"ID":"386e14bc-708a-4130-9b7e-976e11cc5e24","Type":"ContainerStarted","Data":"119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44"} Nov 02 20:04:52.398620 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:52.398573 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:04:52.398989 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:52.398964 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 20:04:52.399102 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:52.399013 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 20:04:52.508472 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:52.508422 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:53.398996 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:53.398960 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 20:04:53.399397 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:53.399024 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 20:04:54.399240 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:54.399158 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 20:04:54.399240 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:54.399210 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 20:04:55.399091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:55.399049 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 20:04:55.399273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:55.399104 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 20:04:56.399019 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:56.398986 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 20:04:56.399195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:56.399036 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 20:04:56.454478 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:56.454439 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:57.398771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:57.398731 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 20:04:57.398991 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:57.398791 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 20:04:58.402219 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:58.402182 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:04:58.402721 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:58.402471 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:04:58.403167 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:58.403153 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:04:58.462597 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:58.462564 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:58.990906 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:58.990852 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:04:59.174328 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:04:59.174295 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:01.358423 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:01.358389 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:02.438255 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.438220 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5"] Nov 02 20:05:02.441997 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.441970 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.492966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.492929 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-982b8a\"/\"kube-root-ca.crt\"" Nov 02 20:05:02.540259 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.540227 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-982b8a\"/\"openshift-service-ca.crt\"" Nov 02 20:05:02.540454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.540430 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-982b8a\"/\"default-dockercfg-qzq75\"" Nov 02 20:05:02.540519 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.540462 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-982b8a\"/\"simulator-w6-982b8a-kserve-self-signed-certs\"" Nov 02 20:05:02.541752 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.541727 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5"] Nov 02 20:05:02.544941 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.544913 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-model-cache\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.545058 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.545037 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94af5da2-6687-4956-97d2-ca8ef68b5854-tls-certs\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.545138 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.545116 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrdh7\" (UniqueName: \"kubernetes.io/projected/94af5da2-6687-4956-97d2-ca8ef68b5854-kube-api-access-nrdh7\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.545202 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.545184 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-tmp-dir\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.545263 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.545247 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-dshm\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.545316 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.545279 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-home\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.589247 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.589184 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:02.646064 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.646024 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdh7\" (UniqueName: \"kubernetes.io/projected/94af5da2-6687-4956-97d2-ca8ef68b5854-kube-api-access-nrdh7\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.646064 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.646066 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-tmp-dir\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.646310 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.646116 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-dshm\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.646310 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.646142 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-home\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.646310 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.646207 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-model-cache\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.646455 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.646311 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94af5da2-6687-4956-97d2-ca8ef68b5854-tls-certs\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.646568 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.646544 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-tmp-dir\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.646632 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.646618 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-home\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.646685 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.646673 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-model-cache\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.648405 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.648382 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-dshm\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.648666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.648646 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94af5da2-6687-4956-97d2-ca8ef68b5854-tls-certs\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.701075 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.700993 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrdh7\" (UniqueName: \"kubernetes.io/projected/94af5da2-6687-4956-97d2-ca8ef68b5854-kube-api-access-nrdh7\") pod \"simulator-w6-982b8a-kserve-7b5d974c5-sw8c5\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:02.754034 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:02.753994 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:03.082680 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:03.082590 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:03.130147 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:03.130110 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5"] Nov 02 20:05:03.138182 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:05:03.138137 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94af5da2_6687_4956_97d2_ca8ef68b5854.slice/crio-48ce66cbb30892a802f89730ebfc5c111836cf12b733cbd29fedbf3237ecc6fa WatchSource:0}: Error finding container 48ce66cbb30892a802f89730ebfc5c111836cf12b733cbd29fedbf3237ecc6fa: Status 404 returned error can't find the container with id 48ce66cbb30892a802f89730ebfc5c111836cf12b733cbd29fedbf3237ecc6fa Nov 02 20:05:03.275743 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:03.275698 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:03.399620 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:03.399530 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" event={"ID":"94af5da2-6687-4956-97d2-ca8ef68b5854","Type":"ContainerStarted","Data":"46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42"} Nov 02 20:05:03.399620 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:03.399585 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" event={"ID":"94af5da2-6687-4956-97d2-ca8ef68b5854","Type":"ContainerStarted","Data":"48ce66cbb30892a802f89730ebfc5c111836cf12b733cbd29fedbf3237ecc6fa"} Nov 02 20:05:03.507945 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:03.507849 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podStartSLOduration=1.507836294 podStartE2EDuration="1.507836294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:03.507184599 +0000 UTC m=+740.314968815" watchObservedRunningTime="2026-11-02 20:05:03.507836294 +0000 UTC m=+740.315620509" Nov 02 20:05:03.754322 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:03.754294 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:03.780611 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:05:03.780576 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:05:05.778162 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:05.778127 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:06.270838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:06.270799 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:06.456572 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:06.456536 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:06.637809 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:06.637719 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:08.768150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:08.768118 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:09.323944 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:09.323859 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:09.575534 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:09.575449 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:11.213376 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:11.213299 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:12.754965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:12.754929 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:12.767495 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:12.767468 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:13.477644 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:13.477613 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:05:14.042086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:14.042045 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:14.612981 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:14.612943 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:14.809645 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:14.809593 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:16.100789 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:16.100746 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:17.141417 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:17.141388 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:17.691614 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:17.691536 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:17.883456 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:17.883421 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:20.953479 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:20.953442 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:25.394720 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.394692 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw"] Nov 02 20:05:25.399550 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.399529 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.450912 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.450853 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw"] Nov 02 20:05:25.496852 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.496816 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"kube-root-ca.crt\"" Nov 02 20:05:25.496852 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.496817 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"openshift-service-ca.crt\"" Nov 02 20:05:25.497466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.497449 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"default-dockercfg-j9ncr\"" Nov 02 20:05:25.497514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.497462 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"simulator-w4-cc64d3-kserve-self-signed-certs\"" Nov 02 20:05:25.558497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.558447 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:25.560604 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.560569 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tls-certs\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.560750 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.560641 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-dshm\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.560750 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.560668 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-model-cache\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.560827 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.560764 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tmp-dir\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.560865 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.560824 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvt84\" (UniqueName: \"kubernetes.io/projected/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-kube-api-access-rvt84\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.560952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.560934 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-home\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.661967 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.661929 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tmp-dir\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.662174 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.662030 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt84\" (UniqueName: \"kubernetes.io/projected/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-kube-api-access-rvt84\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.662174 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.662068 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-home\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.662305 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.662226 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tls-certs\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.662419 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.662396 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-dshm\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.662502 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.662438 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-model-cache\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.662502 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.662397 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tmp-dir\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.662732 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.662701 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-home\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.662851 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.662803 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-model-cache\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.664808 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.664787 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tls-certs\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.665005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.664989 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-dshm\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.716112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.716074 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvt84\" (UniqueName: \"kubernetes.io/projected/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-kube-api-access-rvt84\") pod \"simulator-w4-cc64d3-kserve-555d8c859b-hbthw\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:25.783869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:25.783831 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:26.011403 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:26.011314 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:26.380219 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:26.380137 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw"] Nov 02 20:05:26.387862 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:05:26.387832 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46dbd4e4_a31e_4587_90f2_8bb3ad0832aa.slice/crio-f916385e43ae57e830d6134b155da557c33afb1daeb1a2b8ae1997ab531f9f48 WatchSource:0}: Error finding container f916385e43ae57e830d6134b155da557c33afb1daeb1a2b8ae1997ab531f9f48: Status 404 returned error can't find the container with id f916385e43ae57e830d6134b155da557c33afb1daeb1a2b8ae1997ab531f9f48 Nov 02 20:05:26.526395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:26.526353 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" event={"ID":"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa","Type":"ContainerStarted","Data":"32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172"} Nov 02 20:05:26.526775 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:26.526401 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" event={"ID":"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa","Type":"ContainerStarted","Data":"f916385e43ae57e830d6134b155da557c33afb1daeb1a2b8ae1997ab531f9f48"} Nov 02 20:05:26.637926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:26.637816 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podStartSLOduration=1.6378005789999999 podStartE2EDuration="1.637800579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:26.636250399 +0000 UTC m=+763.444034615" watchObservedRunningTime="2026-11-02 20:05:26.637800579 +0000 UTC m=+763.445584797" Nov 02 20:05:27.011681 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:27.011645 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:28.474746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:28.474708 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:29.065371 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:29.065333 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:29.258297 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:29.258260 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:30.354512 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.354477 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:30.590736 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.590702 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh"] Nov 02 20:05:30.594205 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.594180 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.694034 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.693991 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ae8e6e\"/\"kube-root-ca.crt\"" Nov 02 20:05:30.694204 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.694101 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ae8e6e\"/\"openshift-service-ca.crt\"" Nov 02 20:05:30.694248 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.694225 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ae8e6e\"/\"default-dockercfg-6lc96\"" Nov 02 20:05:30.694308 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.694296 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ae8e6e\"/\"simulator-w5-ae8e6e-kserve-self-signed-certs\"" Nov 02 20:05:30.695642 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.695624 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh"] Nov 02 20:05:30.719638 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.719605 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-tmp-dir\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.719792 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.719641 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-home\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.719792 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.719661 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-model-cache\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.719792 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.719736 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-dshm\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.719792 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.719788 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08b219f6-3577-4242-9bca-448a02c6b524-tls-certs\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.719971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.719833 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2km4l\" (UniqueName: \"kubernetes.io/projected/08b219f6-3577-4242-9bca-448a02c6b524-kube-api-access-2km4l\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.724293 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.724271 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:30.820684 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.820631 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-tmp-dir\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.820684 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.820687 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-home\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.820987 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.820713 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-model-cache\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.820987 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.820748 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-dshm\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.820987 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.820783 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08b219f6-3577-4242-9bca-448a02c6b524-tls-certs\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.820987 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.820849 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2km4l\" (UniqueName: \"kubernetes.io/projected/08b219f6-3577-4242-9bca-448a02c6b524-kube-api-access-2km4l\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.821194 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.821103 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-tmp-dir\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.821194 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.821153 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-model-cache\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.821194 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.821167 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-home\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.823168 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.823146 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-dshm\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.823331 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.823311 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08b219f6-3577-4242-9bca-448a02c6b524-tls-certs\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.875008 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.874978 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2km4l\" (UniqueName: \"kubernetes.io/projected/08b219f6-3577-4242-9bca-448a02c6b524-kube-api-access-2km4l\") pod \"simulator-w5-ae8e6e-kserve-6547b44896-jkzmh\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:30.906254 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:30.906215 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:31.313428 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:31.313353 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh"] Nov 02 20:05:31.316281 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:05:31.316246 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08b219f6_3577_4242_9bca_448a02c6b524.slice/crio-3aa4cf6d3d4afb15093644108af1ce14062ed5fc6de6e2038b319f1df218f972 WatchSource:0}: Error finding container 3aa4cf6d3d4afb15093644108af1ce14062ed5fc6de6e2038b319f1df218f972: Status 404 returned error can't find the container with id 3aa4cf6d3d4afb15093644108af1ce14062ed5fc6de6e2038b319f1df218f972 Nov 02 20:05:31.550015 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:31.549976 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" event={"ID":"08b219f6-3577-4242-9bca-448a02c6b524","Type":"ContainerStarted","Data":"ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18"} Nov 02 20:05:31.550382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:31.550027 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" event={"ID":"08b219f6-3577-4242-9bca-448a02c6b524","Type":"ContainerStarted","Data":"3aa4cf6d3d4afb15093644108af1ce14062ed5fc6de6e2038b319f1df218f972"} Nov 02 20:05:31.659217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:31.659154 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podStartSLOduration=1.659135655 podStartE2EDuration="1.659135655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:31.657761227 +0000 UTC m=+768.465545443" watchObservedRunningTime="2026-11-02 20:05:31.659135655 +0000 UTC m=+768.466919874" Nov 02 20:05:31.906592 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:31.906561 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:33.778374 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:05:33.778338 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:05:34.952604 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:34.952566 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:35.796834 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:35.796797 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:36.011919 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:36.011886 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:36.025130 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:36.025102 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:36.583902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:36.583854 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:05:39.456621 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:39.456587 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:40.907465 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:40.907432 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:40.921003 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:40.920970 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:41.647897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:41.645097 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:05:42.279292 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:42.279261 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:43.962554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:43.962521 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:45.176807 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:45.176772 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:45.443851 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:45.443758 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:48.449449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:48.449413 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:52.955966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:52.955929 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:55.575844 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.575812 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b"] Nov 02 20:05:55.580624 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.580604 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.662543 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.662502 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b"] Nov 02 20:05:55.678209 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.678174 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"default-dockercfg-sqlwk\"" Nov 02 20:05:55.678382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.678239 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"openshift-service-ca.crt\"" Nov 02 20:05:55.678382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.678290 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"simulator-w1-accdf5-kserve-self-signed-certs\"" Nov 02 20:05:55.678382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.678323 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"kube-root-ca.crt\"" Nov 02 20:05:55.679742 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.679726 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e29f5a7-5aa5-441e-befd-42f50aefefde-tls-certs\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.679807 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.679755 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5272\" (UniqueName: \"kubernetes.io/projected/8e29f5a7-5aa5-441e-befd-42f50aefefde-kube-api-access-c5272\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.679807 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.679778 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-tmp-dir\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.679928 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.679853 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-model-cache\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.679982 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.679961 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-dshm\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.680075 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.680063 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-home\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.721038 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.720996 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:05:55.780773 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.780736 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e29f5a7-5aa5-441e-befd-42f50aefefde-tls-certs\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.780996 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.780789 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5272\" (UniqueName: \"kubernetes.io/projected/8e29f5a7-5aa5-441e-befd-42f50aefefde-kube-api-access-c5272\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.780996 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.780829 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-tmp-dir\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.780996 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.780893 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-model-cache\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.780996 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.780919 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-dshm\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.780996 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.780979 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-home\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.781443 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.781411 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-tmp-dir\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.781443 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.781439 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-model-cache\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.781597 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.781459 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-home\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.783214 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.783194 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-dshm\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.783449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.783431 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e29f5a7-5aa5-441e-befd-42f50aefefde-tls-certs\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.835155 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.835083 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5272\" (UniqueName: \"kubernetes.io/projected/8e29f5a7-5aa5-441e-befd-42f50aefefde-kube-api-access-c5272\") pod \"simulator-w1-accdf5-kserve-67d8987bb6-mss6b\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:55.893497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:55.893458 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:56.269459 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:56.269425 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b"] Nov 02 20:05:56.276916 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:05:56.276870 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e29f5a7_5aa5_441e_befd_42f50aefefde.slice/crio-d8cff6b313d9c5b61b2c4461c6327f8b95418bac4444c9b521b6c39ac65913da WatchSource:0}: Error finding container d8cff6b313d9c5b61b2c4461c6327f8b95418bac4444c9b521b6c39ac65913da: Status 404 returned error can't find the container with id d8cff6b313d9c5b61b2c4461c6327f8b95418bac4444c9b521b6c39ac65913da Nov 02 20:05:56.691743 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:56.691707 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" event={"ID":"8e29f5a7-5aa5-441e-befd-42f50aefefde","Type":"ContainerStarted","Data":"22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5"} Nov 02 20:05:56.692216 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:56.691761 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" event={"ID":"8e29f5a7-5aa5-441e-befd-42f50aefefde","Type":"ContainerStarted","Data":"d8cff6b313d9c5b61b2c4461c6327f8b95418bac4444c9b521b6c39ac65913da"} Nov 02 20:05:56.799476 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:56.799427 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podStartSLOduration=1.799414836 podStartE2EDuration="1.799414836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:56.798749215 +0000 UTC m=+793.606533449" watchObservedRunningTime="2026-11-02 20:05:56.799414836 +0000 UTC m=+793.607199051" Nov 02 20:05:56.894302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:56.894272 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:05:57.473573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:05:57.473540 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:01.992578 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:01.992542 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:03.779626 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:06:03.779584 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:06:05.894402 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:05.894366 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:06:05.906923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:05.906893 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:06:06.466159 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:06.466127 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:06.749926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:06.749840 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:06:09.942549 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:09.942508 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:10.956514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:10.956474 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:15.447472 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:15.447434 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:18.909319 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:18.909282 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:20.082390 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.082357 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:20.449540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.449506 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt"] Nov 02 20:06:20.453517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.453499 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.527598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.527563 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-home\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.527764 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.527632 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-dshm\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.527764 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.527698 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/724fade4-5d2f-4d3d-9952-351ab691d309-tls-certs\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.527889 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.527771 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-tmp-dir\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.527889 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.527796 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj2wj\" (UniqueName: \"kubernetes.io/projected/724fade4-5d2f-4d3d-9952-351ab691d309-kube-api-access-hj2wj\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.527998 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.527975 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-model-cache\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.550948 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.550912 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"openshift-service-ca.crt\"" Nov 02 20:06:20.551058 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.550953 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"kube-root-ca.crt\"" Nov 02 20:06:20.551157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.551138 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"default-dockercfg-gpd7p\"" Nov 02 20:06:20.551262 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.551246 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"simulator-w2-9fcedd-kserve-self-signed-certs\"" Nov 02 20:06:20.552516 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.552497 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt"] Nov 02 20:06:20.629473 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.629440 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-model-cache\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.629666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.629499 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-home\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.629666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.629538 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-dshm\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.629666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.629560 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/724fade4-5d2f-4d3d-9952-351ab691d309-tls-certs\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.629666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.629583 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-tmp-dir\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.629666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.629605 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj2wj\" (UniqueName: \"kubernetes.io/projected/724fade4-5d2f-4d3d-9952-351ab691d309-kube-api-access-hj2wj\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.629996 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.629915 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-model-cache\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.629996 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.629979 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-home\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.630199 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.630178 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-tmp-dir\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.631917 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.631897 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-dshm\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.632193 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.632171 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/724fade4-5d2f-4d3d-9952-351ab691d309-tls-certs\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.683689 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.683658 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj2wj\" (UniqueName: \"kubernetes.io/projected/724fade4-5d2f-4d3d-9952-351ab691d309-kube-api-access-hj2wj\") pod \"simulator-w2-9fcedd-kserve-5745db98fb-vk5rt\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:20.796383 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:20.796289 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:21.167845 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:21.167807 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt"] Nov 02 20:06:21.175648 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:06:21.175619 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod724fade4_5d2f_4d3d_9952_351ab691d309.slice/crio-189a1e075dd2e21ee88006996ad0bd604e2b5a8f796d29150e7f83af263adeb9 WatchSource:0}: Error finding container 189a1e075dd2e21ee88006996ad0bd604e2b5a8f796d29150e7f83af263adeb9: Status 404 returned error can't find the container with id 189a1e075dd2e21ee88006996ad0bd604e2b5a8f796d29150e7f83af263adeb9 Nov 02 20:06:21.813276 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:21.813222 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" event={"ID":"724fade4-5d2f-4d3d-9952-351ab691d309","Type":"ContainerStarted","Data":"81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a"} Nov 02 20:06:21.813276 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:21.813284 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" event={"ID":"724fade4-5d2f-4d3d-9952-351ab691d309","Type":"ContainerStarted","Data":"189a1e075dd2e21ee88006996ad0bd604e2b5a8f796d29150e7f83af263adeb9"} Nov 02 20:06:21.923930 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:21.923854 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podStartSLOduration=1.9238396180000001 podStartE2EDuration="1.923839618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:21.92266539 +0000 UTC m=+818.730449691" watchObservedRunningTime="2026-11-02 20:06:21.923839618 +0000 UTC m=+818.731623842" Nov 02 20:06:22.214655 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:22.214620 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:22.796910 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:22.796858 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:24.450288 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:24.450201 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:29.024611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:29.024571 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:30.796578 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:30.796547 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:30.809276 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:30.809249 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:30.875236 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:30.875211 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:06:31.036896 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.036848 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x"] Nov 02 20:06:31.040726 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.040705 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.090554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.090525 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"premium-w4-cc64d3-kserve-self-signed-certs\"" Nov 02 20:06:31.128951 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.128909 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x"] Nov 02 20:06:31.133613 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.133576 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-model-cache\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.133758 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.133630 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-dshm\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.133758 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.133738 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0475c8c4-b045-4faf-b21f-6501ce60a3df-tls-certs\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.133869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.133782 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-home\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.133922 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.133899 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds56t\" (UniqueName: \"kubernetes.io/projected/0475c8c4-b045-4faf-b21f-6501ce60a3df-kube-api-access-ds56t\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.134003 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.133991 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-tmp-dir\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.168352 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.168268 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:31.234946 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.234907 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-tmp-dir\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.234946 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.234952 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-model-cache\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.235200 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.234976 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-dshm\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.235200 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.235011 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0475c8c4-b045-4faf-b21f-6501ce60a3df-tls-certs\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.235200 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.235041 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-home\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.235200 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.235084 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds56t\" (UniqueName: \"kubernetes.io/projected/0475c8c4-b045-4faf-b21f-6501ce60a3df-kube-api-access-ds56t\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.235473 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.235362 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-tmp-dir\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.235473 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.235424 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-model-cache\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.235581 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.235518 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-home\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.237414 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.237391 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-dshm\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.237782 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.237753 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0475c8c4-b045-4faf-b21f-6501ce60a3df-tls-certs\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.293913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.293837 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds56t\" (UniqueName: \"kubernetes.io/projected/0475c8c4-b045-4faf-b21f-6501ce60a3df-kube-api-access-ds56t\") pod \"premium-w4-cc64d3-kserve-599c7c4c7-lrr9x\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.355715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.355627 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:31.740351 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.740319 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x"] Nov 02 20:06:31.748424 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:06:31.748394 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0475c8c4_b045_4faf_b21f_6501ce60a3df.slice/crio-6df4c28c0646421cfdbc0f8bf9b228dc1d35cf686d44fcc0d11de95399239e5f WatchSource:0}: Error finding container 6df4c28c0646421cfdbc0f8bf9b228dc1d35cf686d44fcc0d11de95399239e5f: Status 404 returned error can't find the container with id 6df4c28c0646421cfdbc0f8bf9b228dc1d35cf686d44fcc0d11de95399239e5f Nov 02 20:06:31.870558 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.870518 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" event={"ID":"0475c8c4-b045-4faf-b21f-6501ce60a3df","Type":"ContainerStarted","Data":"b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b"} Nov 02 20:06:31.870558 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.870559 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" event={"ID":"0475c8c4-b045-4faf-b21f-6501ce60a3df","Type":"ContainerStarted","Data":"6df4c28c0646421cfdbc0f8bf9b228dc1d35cf686d44fcc0d11de95399239e5f"} Nov 02 20:06:31.979809 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:31.979753 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podStartSLOduration=1.97973856 podStartE2EDuration="1.97973856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:31.978914556 +0000 UTC m=+828.786698775" watchObservedRunningTime="2026-11-02 20:06:31.97973856 +0000 UTC m=+828.787522775" Nov 02 20:06:32.356726 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:32.356691 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:32.833715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:32.833685 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:33.005770 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:33.005729 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:33.508933 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:33.508862 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:33.786402 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:06:33.786324 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:06:37.970190 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:37.970146 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:41.356653 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:41.356619 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:41.369144 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:41.369118 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:41.934521 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:41.934494 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:06:42.439386 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:42.439350 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:45.476387 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:45.476346 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:46.336336 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:46.336298 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:46.949651 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:46.949616 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:51.458145 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:51.458107 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:56.050439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:56.050398 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:06:59.182833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:06:59.182790 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:00.552460 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:00.552404 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:03.778772 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:07:03.778738 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:07:05.041910 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:05.041845 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:07.041451 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:07.041416 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:09.560077 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:09.560034 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:14.053981 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:14.053910 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:18.667449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:18.667400 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:19.446444 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:19.446411 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:23.240645 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:23.240602 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:27.763974 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:27.763924 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:32.248980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:32.248945 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:33.780303 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:07:33.780265 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:07:36.871215 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:36.871174 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:38.131802 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:38.131770 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:41.384600 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:41.384564 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:43.332909 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.330787 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm"] Nov 02 20:07:43.337536 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.335600 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.389899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.389430 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm"] Nov 02 20:07:43.432668 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.432637 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"premium-w2-9fcedd-kserve-self-signed-certs\"" Nov 02 20:07:43.482532 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.482494 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:43.505927 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.505890 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-dshm\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.505927 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.505919 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390dcf22-a018-4320-986a-9b45f5958a70-tls-certs\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.506176 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.506028 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-tmp-dir\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.506176 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.506097 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-model-cache\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.506176 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.506172 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-home\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.506327 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.506256 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28z7p\" (UniqueName: \"kubernetes.io/projected/390dcf22-a018-4320-986a-9b45f5958a70-kube-api-access-28z7p\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.607365 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.607277 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-model-cache\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.607365 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.607346 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-home\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.607600 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.607409 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28z7p\" (UniqueName: \"kubernetes.io/projected/390dcf22-a018-4320-986a-9b45f5958a70-kube-api-access-28z7p\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.607600 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.607461 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-dshm\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.607600 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.607484 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390dcf22-a018-4320-986a-9b45f5958a70-tls-certs\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.607600 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.607575 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-tmp-dir\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.607816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.607791 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-model-cache\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.607901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.607830 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-home\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.608000 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.607983 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-tmp-dir\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.610234 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.610210 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390dcf22-a018-4320-986a-9b45f5958a70-tls-certs\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.610502 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.610470 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-dshm\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.665568 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.665536 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28z7p\" (UniqueName: \"kubernetes.io/projected/390dcf22-a018-4320-986a-9b45f5958a70-kube-api-access-28z7p\") pod \"premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:43.953468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:43.953432 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:44.329220 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:44.329129 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm"] Nov 02 20:07:44.337482 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:07:44.337451 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod390dcf22_a018_4320_986a_9b45f5958a70.slice/crio-31d0ca1f18099c3de3cc8dc5ac6748f8040e7be7edbd8776d854636080aeead5 WatchSource:0}: Error finding container 31d0ca1f18099c3de3cc8dc5ac6748f8040e7be7edbd8776d854636080aeead5: Status 404 returned error can't find the container with id 31d0ca1f18099c3de3cc8dc5ac6748f8040e7be7edbd8776d854636080aeead5 Nov 02 20:07:45.235533 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:45.235477 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" event={"ID":"390dcf22-a018-4320-986a-9b45f5958a70","Type":"ContainerStarted","Data":"8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954"} Nov 02 20:07:45.235847 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:45.235549 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" event={"ID":"390dcf22-a018-4320-986a-9b45f5958a70","Type":"ContainerStarted","Data":"31d0ca1f18099c3de3cc8dc5ac6748f8040e7be7edbd8776d854636080aeead5"} Nov 02 20:07:45.345942 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:45.345860 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podStartSLOduration=2.345847853 podStartE2EDuration="2.345847853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:45.345518119 +0000 UTC m=+902.153302345" watchObservedRunningTime="2026-11-02 20:07:45.345847853 +0000 UTC m=+902.153632068" Nov 02 20:07:45.815899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:45.815448 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:45.954155 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:45.954109 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:45.990893 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:45.990856 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:48.979504 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:48.979470 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:49.682226 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:49.682172 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:50.461122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:50.461022 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:51.072229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:51.072197 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:51.243140 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:51.243108 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:53.954668 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:53.954588 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:53.967074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:53.967050 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:54.292194 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:54.292120 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:07:54.960354 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:54.960315 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:07:59.507171 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:07:59.507135 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:03.779611 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:08:03.779576 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:08:05.091797 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:05.091758 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:08.775772 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:08.775735 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:13.271498 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:13.271466 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:17.861949 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:17.861910 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:22.457454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:22.457423 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:26.965161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:26.965122 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:31.475931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:31.475869 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:33.778972 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:08:33.778930 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:08:35.991617 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:35.991579 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:40.482439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:40.482400 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:44.970839 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:44.970789 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:46.143684 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:46.143642 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:46.447390 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:46.447356 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:49.552626 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:49.552590 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:54.062366 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:54.062327 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:08:58.674722 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:08:58.674678 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:02.292117 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:02.292070 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:03.187800 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:03.187768 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:03.779311 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:09:03.779279 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:09:07.755526 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:07.755494 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:12.349272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:12.349228 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:16.878220 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:16.878181 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:21.568673 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:21.568633 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:26.117052 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:26.116958 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:30.583181 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:30.583142 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:33.785133 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:09:33.785093 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:09:35.306050 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:35.306012 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:37.585157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:37.585119 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:37.766616 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:37.766587 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:39.969235 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:39.969166 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:44.475602 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:44.475568 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:46.459485 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:46.459449 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:48.953204 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:48.953169 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:53.477835 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:53.477800 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:54.568930 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.568009 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p"] Nov 02 20:09:54.573803 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.573765 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.625199 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.625166 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p"] Nov 02 20:09:54.670727 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.670691 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-982b8a\"/\"distinct-w6-982b8a-kserve-self-signed-certs\"" Nov 02 20:09:54.683194 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.683166 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfpx\" (UniqueName: \"kubernetes.io/projected/787327eb-1c48-4383-b74c-7d468b0cb511-kube-api-access-mpfpx\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.683312 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.683212 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787327eb-1c48-4383-b74c-7d468b0cb511-tls-certs\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.683312 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.683278 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-dshm\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.683399 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.683384 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-model-cache\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.683433 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.683406 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-home\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.683511 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.683498 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-tmp-dir\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.708633 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.708595 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:09:54.784590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.784553 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-tmp-dir\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.784762 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.784653 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfpx\" (UniqueName: \"kubernetes.io/projected/787327eb-1c48-4383-b74c-7d468b0cb511-kube-api-access-mpfpx\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.784762 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.784695 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787327eb-1c48-4383-b74c-7d468b0cb511-tls-certs\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.784762 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.784720 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-dshm\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.784948 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.784782 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-model-cache\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.784948 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.784804 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-home\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.785188 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.785161 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-tmp-dir\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.785272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.785218 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-home\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.785329 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.785279 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-model-cache\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.787149 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.787130 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-dshm\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.787551 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.787523 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787327eb-1c48-4383-b74c-7d468b0cb511-tls-certs\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.843604 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.843514 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfpx\" (UniqueName: \"kubernetes.io/projected/787327eb-1c48-4383-b74c-7d468b0cb511-kube-api-access-mpfpx\") pod \"distinct-w6-982b8a-kserve-6cf454f54f-hph4p\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:54.886074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:54.886040 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:55.261000 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:55.260964 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p"] Nov 02 20:09:55.268805 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:09:55.268779 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod787327eb_1c48_4383_b74c_7d468b0cb511.slice/crio-e8a81d727743de27940d2712d7ed3140dc8042233c72b9fdfe8a80100cdaf158 WatchSource:0}: Error finding container e8a81d727743de27940d2712d7ed3140dc8042233c72b9fdfe8a80100cdaf158: Status 404 returned error can't find the container with id e8a81d727743de27940d2712d7ed3140dc8042233c72b9fdfe8a80100cdaf158 Nov 02 20:09:55.951108 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:55.951073 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" event={"ID":"787327eb-1c48-4383-b74c-7d468b0cb511","Type":"ContainerStarted","Data":"1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44"} Nov 02 20:09:55.951584 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:55.951116 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" event={"ID":"787327eb-1c48-4383-b74c-7d468b0cb511","Type":"ContainerStarted","Data":"e8a81d727743de27940d2712d7ed3140dc8042233c72b9fdfe8a80100cdaf158"} Nov 02 20:09:56.061517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:56.061463 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podStartSLOduration=2.061448189 podStartE2EDuration="2.061448189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:56.060980574 +0000 UTC m=+1032.868764810" watchObservedRunningTime="2026-11-02 20:09:56.061448189 +0000 UTC m=+1032.869232408" Nov 02 20:09:56.886637 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:56.886605 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:09:57.979891 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:09:57.979845 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:01.474657 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.474621 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk"] Nov 02 20:10:01.478153 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.478135 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.529357 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.529316 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-035e47-openshift-default-dockercfg-rx8rw\"" Nov 02 20:10:01.554139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.554096 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.554270 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.554212 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.554270 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.554242 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8gzq\" (UniqueName: \"kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-kube-api-access-r8gzq\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.554361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.554343 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.554402 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.554379 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.554436 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.554410 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.554503 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.554481 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9874e15-3cc4-4b17-962a-94bf2d026d94-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.554535 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.554526 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.554570 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.554545 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.578859 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.578825 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk"] Nov 02 20:10:01.655657 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.655615 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.655838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.655686 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.655838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.655710 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8gzq\" (UniqueName: \"kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-kube-api-access-r8gzq\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.655838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.655749 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.655838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.655774 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.655838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.655814 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.656119 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.655843 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9874e15-3cc4-4b17-962a-94bf2d026d94-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.656119 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.655901 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.656119 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.655926 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.656119 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.656067 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.656659 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.656631 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.656779 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.656684 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.656844 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.656788 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9874e15-3cc4-4b17-962a-94bf2d026d94-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.657081 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.657047 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.658838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.658814 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.659143 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.659120 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.710896 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.710844 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.711172 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.711148 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8gzq\" (UniqueName: \"kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-kube-api-access-r8gzq\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:01.792476 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:01.792382 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:02.220619 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:02.220583 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk"] Nov 02 20:10:02.228350 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:10:02.228321 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9874e15_3cc4_4b17_962a_94bf2d026d94.slice/crio-9cc252836967006764ab30160ab7800cc3d9b9421ace40285805ef938fec8061 WatchSource:0}: Error finding container 9cc252836967006764ab30160ab7800cc3d9b9421ace40285805ef938fec8061: Status 404 returned error can't find the container with id 9cc252836967006764ab30160ab7800cc3d9b9421ace40285805ef938fec8061 Nov 02 20:10:02.230530 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:02.230486 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:10:02.230602 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:02.230565 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:10:02.230602 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:02.230596 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:10:02.343968 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:02.343911 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:02.546435 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:02.546338 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:02.993275 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:02.993237 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" event={"ID":"a9874e15-3cc4-4b17-962a-94bf2d026d94","Type":"ContainerStarted","Data":"f104acf0f5f2761d809fb129a1a2220a421de38e4bea3a661c06e57e530f50b8"} Nov 02 20:10:02.993491 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:02.993302 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" event={"ID":"a9874e15-3cc4-4b17-962a-94bf2d026d94","Type":"ContainerStarted","Data":"9cc252836967006764ab30160ab7800cc3d9b9421ace40285805ef938fec8061"} Nov 02 20:10:03.780320 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:10:03.780278 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:10:03.793547 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:03.793519 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:03.794855 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:03.794826 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:10:03.794999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:03.794898 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" podUID="a9874e15-3cc4-4b17-962a-94bf2d026d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 20:10:04.793658 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:04.793626 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:10:04.794096 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:04.793678 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" podUID="a9874e15-3cc4-4b17-962a-94bf2d026d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 20:10:04.886218 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:04.886180 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:10:04.898903 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:04.898857 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:10:05.012314 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:05.012261 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" podStartSLOduration=4.012247892 podStartE2EDuration="4.012247892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:03.154082573 +0000 UTC m=+1039.961866817" watchObservedRunningTime="2026-11-02 20:10:05.012247892 +0000 UTC m=+1041.820032110" Nov 02 20:10:05.015450 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:05.015427 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:10:05.621995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:05.621961 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:05.793072 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:05.793031 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:10:05.793251 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:05.793105 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" podUID="a9874e15-3cc4-4b17-962a-94bf2d026d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 20:10:06.793454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:06.793413 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:10:06.794021 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:06.793494 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" podUID="a9874e15-3cc4-4b17-962a-94bf2d026d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 20:10:07.054563 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:07.054434 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:07.793584 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:07.793547 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:10:07.794029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:07.793599 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" podUID="a9874e15-3cc4-4b17-962a-94bf2d026d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 20:10:08.967271 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:08.967241 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:08.967837 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:08.967661 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:08.968748 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:08.968724 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:10:11.663958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:11.663920 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:16.183732 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.183689 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:16.299764 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.299724 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx"] Nov 02 20:10:16.303416 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.303393 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.353282 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.353251 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-accdf5-openshift-default-dockercfg-c8k9m\"" Nov 02 20:10:16.403664 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.403625 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx"] Nov 02 20:10:16.412221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.412185 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.412221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.412221 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.412497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.412246 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fb1be7a-a4de-40ee-9805-dac41de74c30-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.412497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.412345 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.412497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.412397 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.412497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.412477 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.412725 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.412555 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwqg2\" (UniqueName: \"kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-kube-api-access-jwqg2\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.412725 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.412607 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.412725 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.412631 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514188 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514096 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514188 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514165 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514417 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514206 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwqg2\" (UniqueName: \"kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-kube-api-access-jwqg2\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514417 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514255 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514417 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514285 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514417 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514359 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514417 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514385 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514417 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514414 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fb1be7a-a4de-40ee-9805-dac41de74c30-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514737 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514477 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514737 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514558 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514737 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514636 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514842 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514790 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.514988 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.514964 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.515159 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.515143 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fb1be7a-a4de-40ee-9805-dac41de74c30-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.516608 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.516586 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.516740 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.516720 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.568749 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.568715 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.568915 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.568803 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwqg2\" (UniqueName: \"kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-kube-api-access-jwqg2\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:16.616420 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:16.616388 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:17.047051 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:17.047009 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx"] Nov 02 20:10:17.054823 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:10:17.054792 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb1be7a_a4de_40ee_9805_dac41de74c30.slice/crio-57468a942ffc3d275e95968842ff52f16a3c2a15dea4638778f8cec76cb6729a WatchSource:0}: Error finding container 57468a942ffc3d275e95968842ff52f16a3c2a15dea4638778f8cec76cb6729a: Status 404 returned error can't find the container with id 57468a942ffc3d275e95968842ff52f16a3c2a15dea4638778f8cec76cb6729a Nov 02 20:10:17.057187 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:17.057151 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:10:17.057262 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:17.057219 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:10:17.057262 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:17.057250 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:10:17.061936 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:17.061905 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" event={"ID":"0fb1be7a-a4de-40ee-9805-dac41de74c30","Type":"ContainerStarted","Data":"57468a942ffc3d275e95968842ff52f16a3c2a15dea4638778f8cec76cb6729a"} Nov 02 20:10:18.068069 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:18.068029 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" event={"ID":"0fb1be7a-a4de-40ee-9805-dac41de74c30","Type":"ContainerStarted","Data":"55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103"} Nov 02 20:10:18.272646 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:18.272596 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" podStartSLOduration=2.272583564 podStartE2EDuration="2.272583564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:18.27242566 +0000 UTC m=+1055.080209896" watchObservedRunningTime="2026-11-02 20:10:18.272583564 +0000 UTC m=+1055.080367781" Nov 02 20:10:18.616985 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:18.616923 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:18.618739 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:18.618715 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 20:10:18.618852 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:18.618773 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" podUID="0fb1be7a-a4de-40ee-9805-dac41de74c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 20:10:19.617456 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:19.617421 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 20:10:19.617891 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:19.617473 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" podUID="0fb1be7a-a4de-40ee-9805-dac41de74c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 20:10:20.617676 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:20.617639 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 20:10:20.618095 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:20.617709 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" podUID="0fb1be7a-a4de-40ee-9805-dac41de74c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 20:10:20.668143 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:20.668102 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:21.617655 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:21.617621 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 20:10:21.617836 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:21.617679 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" podUID="0fb1be7a-a4de-40ee-9805-dac41de74c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 20:10:22.617405 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:22.617365 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 20:10:22.617597 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:22.617456 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" podUID="0fb1be7a-a4de-40ee-9805-dac41de74c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 20:10:23.369715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:23.369675 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:23.544678 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:23.544621 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:23.746525 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:23.746492 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:23.747661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:23.747630 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:23.748113 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:23.748094 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:10:25.164317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:25.164275 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:29.662478 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:29.662436 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:33.779366 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:10:33.779326 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:10:34.262140 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:34.262105 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:35.340174 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.340132 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8"] Nov 02 20:10:35.344508 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.344483 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.396579 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.396542 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8"] Nov 02 20:10:35.440128 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.440096 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-model-cache\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.440128 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.440134 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-tmp-dir\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.440349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.440154 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41b6de3b-682d-4a07-9297-e67df9f94243-tls-certs\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.440349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.440263 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-dshm\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.440349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.440301 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-home\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.440452 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.440406 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw2qb\" (UniqueName: \"kubernetes.io/projected/41b6de3b-682d-4a07-9297-e67df9f94243-kube-api-access-cw2qb\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.441120 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.441104 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ae8e6e\"/\"distinct-w5-ae8e6e-kserve-self-signed-certs\"" Nov 02 20:10:35.496102 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.496067 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:35.541554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.541508 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-model-cache\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.541554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.541554 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-tmp-dir\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.541818 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.541573 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41b6de3b-682d-4a07-9297-e67df9f94243-tls-certs\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.541818 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.541596 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-dshm\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.541818 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.541752 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-home\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.542009 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.541851 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw2qb\" (UniqueName: \"kubernetes.io/projected/41b6de3b-682d-4a07-9297-e67df9f94243-kube-api-access-cw2qb\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.542182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.542037 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-tmp-dir\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.542182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.542051 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-model-cache\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.542370 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.542346 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-home\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.544284 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.544256 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-dshm\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.544449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.544431 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41b6de3b-682d-4a07-9297-e67df9f94243-tls-certs\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.596518 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.596438 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw2qb\" (UniqueName: \"kubernetes.io/projected/41b6de3b-682d-4a07-9297-e67df9f94243-kube-api-access-cw2qb\") pod \"distinct-w5-ae8e6e-kserve-64594bdd96-49hq8\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:35.659431 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:35.659393 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:36.055595 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:36.055554 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8"] Nov 02 20:10:36.062820 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:10:36.062794 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41b6de3b_682d_4a07_9297_e67df9f94243.slice/crio-d8cdae0b70e88bf2c440bd06fa1ece73ab5853f864b9107cdc00e990206d27f6 WatchSource:0}: Error finding container d8cdae0b70e88bf2c440bd06fa1ece73ab5853f864b9107cdc00e990206d27f6: Status 404 returned error can't find the container with id d8cdae0b70e88bf2c440bd06fa1ece73ab5853f864b9107cdc00e990206d27f6 Nov 02 20:10:36.161400 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:36.161365 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" event={"ID":"41b6de3b-682d-4a07-9297-e67df9f94243","Type":"ContainerStarted","Data":"cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b"} Nov 02 20:10:36.161400 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:36.161414 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" event={"ID":"41b6de3b-682d-4a07-9297-e67df9f94243","Type":"ContainerStarted","Data":"d8cdae0b70e88bf2c440bd06fa1ece73ab5853f864b9107cdc00e990206d27f6"} Nov 02 20:10:36.270109 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:36.270058 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podStartSLOduration=1.270042285 podStartE2EDuration="1.270042285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:36.269503661 +0000 UTC m=+1073.077287896" watchObservedRunningTime="2026-11-02 20:10:36.270042285 +0000 UTC m=+1073.077826501" Nov 02 20:10:36.659754 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:36.659722 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:38.762862 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:38.762827 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:43.306207 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:43.306167 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:45.660190 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:45.660153 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:45.673146 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:45.673113 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:46.221814 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:46.221771 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:10:46.803358 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:46.803326 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:47.083040 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:47.082957 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:47.263419 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:47.263385 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:47.859231 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:47.859196 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:52.354660 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:52.354626 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:53.001094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:53.001055 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:56.981194 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:56.981109 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:10:58.031125 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:10:58.031058 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:00.632456 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.632421 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z"] Nov 02 20:11:00.636049 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.636033 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.686283 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.686246 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ae8e6e-openshift-default-dockercfg-4p7q9\"" Nov 02 20:11:00.708766 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.708729 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8w7q\" (UniqueName: \"kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-kube-api-access-m8w7q\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.708955 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.708775 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.708955 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.708836 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.709029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.708972 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39528d6-5e81-437f-b049-79e42cc4f6c4-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.709029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.709002 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.709029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.709023 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.709136 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.709038 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.709177 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.709128 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.709211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.709174 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.735747 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.735714 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z"] Nov 02 20:11:00.810387 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.810354 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.810576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.810405 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8w7q\" (UniqueName: \"kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-kube-api-access-m8w7q\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.810576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.810433 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.810576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.810448 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.810576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.810499 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39528d6-5e81-437f-b049-79e42cc4f6c4-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.810576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.810531 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.810576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.810563 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.810944 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.810660 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.810944 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.810714 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.810944 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.810771 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.810944 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.810833 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.811346 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.811314 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39528d6-5e81-437f-b049-79e42cc4f6c4-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.811495 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.811476 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.811561 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.811493 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.813222 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.813197 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.813304 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.813255 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.865495 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.865463 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.865666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.865615 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8w7q\" (UniqueName: \"kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-kube-api-access-m8w7q\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:00.949900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:00.949850 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:01.386955 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:01.386851 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z"] Nov 02 20:11:01.393581 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:11:01.393547 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39528d6_5e81_437f_b049_79e42cc4f6c4.slice/crio-26618d649b6dc715ccf2987363ff9f7a8ecc2cafe5fa2bab8c51b7b16562231f WatchSource:0}: Error finding container 26618d649b6dc715ccf2987363ff9f7a8ecc2cafe5fa2bab8c51b7b16562231f: Status 404 returned error can't find the container with id 26618d649b6dc715ccf2987363ff9f7a8ecc2cafe5fa2bab8c51b7b16562231f Nov 02 20:11:01.396100 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:01.396051 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:11:01.396221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:01.396139 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:11:01.396221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:01.396188 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:11:01.472216 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:01.472183 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:02.314901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:02.313029 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" event={"ID":"b39528d6-5e81-437f-b049-79e42cc4f6c4","Type":"ContainerStarted","Data":"c562c9b6e48e55f9f7acf2da9a1dacc818946397464f07b43f0456748395168e"} Nov 02 20:11:02.314901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:02.313089 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" event={"ID":"b39528d6-5e81-437f-b049-79e42cc4f6c4","Type":"ContainerStarted","Data":"26618d649b6dc715ccf2987363ff9f7a8ecc2cafe5fa2bab8c51b7b16562231f"} Nov 02 20:11:02.374913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:02.372407 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s"] Nov 02 20:11:02.374913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:02.372660 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" containerID="cri-o://b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b" gracePeriod=30 Nov 02 20:11:02.520214 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:02.517974 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" podStartSLOduration=2.517954653 podStartE2EDuration="2.517954653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:02.516412264 +0000 UTC m=+1099.324196521" watchObservedRunningTime="2026-11-02 20:11:02.517954653 +0000 UTC m=+1099.325738869" Nov 02 20:11:02.953912 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:02.953166 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:02.955409 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:02.955378 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 20:11:02.955645 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:02.955622 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" podUID="b39528d6-5e81-437f-b049-79e42cc4f6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 20:11:03.790409 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:11:03.790371 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:11:03.950899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:03.950308 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 20:11:03.950899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:03.950372 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" podUID="b39528d6-5e81-437f-b049-79e42cc4f6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 20:11:04.950899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:04.950279 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 20:11:04.950899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:04.950340 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" podUID="b39528d6-5e81-437f-b049-79e42cc4f6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 20:11:05.953348 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:05.953272 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 20:11:05.953348 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:05.953334 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" podUID="b39528d6-5e81-437f-b049-79e42cc4f6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 20:11:05.974100 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:05.974064 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:06.953329 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:06.953285 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 20:11:06.953570 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:06.953350 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" podUID="b39528d6-5e81-437f-b049-79e42cc4f6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 20:11:07.571073 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.571046 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:11:07.599755 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.599720 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-envoy\") pod \"8e64aba5-a055-4e97-8a83-dc814e961e89\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " Nov 02 20:11:07.599948 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.599769 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-kube-api-access-b49x4\" (UniqueName: \"kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-kube-api-access-b49x4\") pod \"8e64aba5-a055-4e97-8a83-dc814e961e89\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " Nov 02 20:11:07.599948 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.599800 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e64aba5-a055-4e97-8a83-dc814e961e89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e64aba5-a055-4e97-8a83-dc814e961e89-istiod-ca-cert\") pod \"8e64aba5-a055-4e97-8a83-dc814e961e89\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " Nov 02 20:11:07.599948 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.599838 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-certs\") pod \"8e64aba5-a055-4e97-8a83-dc814e961e89\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " Nov 02 20:11:07.600070 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.599965 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-socket\") pod \"8e64aba5-a055-4e97-8a83-dc814e961e89\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " Nov 02 20:11:07.600070 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600007 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-token\" (UniqueName: \"kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-token\") pod \"8e64aba5-a055-4e97-8a83-dc814e961e89\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " Nov 02 20:11:07.600070 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600041 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-data\") pod \"8e64aba5-a055-4e97-8a83-dc814e961e89\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " Nov 02 20:11:07.600229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600116 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-credential-socket\") pod \"8e64aba5-a055-4e97-8a83-dc814e961e89\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " Nov 02 20:11:07.600229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600173 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-podinfo\") pod \"8e64aba5-a055-4e97-8a83-dc814e961e89\" (UID: \"8e64aba5-a055-4e97-8a83-dc814e961e89\") " Nov 02 20:11:07.600360 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600327 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e64aba5-a055-4e97-8a83-dc814e961e89-istiod-ca-cert" pod "8e64aba5-a055-4e97-8a83-dc814e961e89" (UID: "8e64aba5-a055-4e97-8a83-dc814e961e89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:11:07.600608 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600429 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-socket" pod "8e64aba5-a055-4e97-8a83-dc814e961e89" (UID: "8e64aba5-a055-4e97-8a83-dc814e961e89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:11:07.600608 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600474 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-data" pod "8e64aba5-a055-4e97-8a83-dc814e961e89" (UID: "8e64aba5-a055-4e97-8a83-dc814e961e89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:11:07.600608 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600553 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-certs" pod "8e64aba5-a055-4e97-8a83-dc814e961e89" (UID: "8e64aba5-a055-4e97-8a83-dc814e961e89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:11:07.600821 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600649 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-credential-socket" pod "8e64aba5-a055-4e97-8a83-dc814e961e89" (UID: "8e64aba5-a055-4e97-8a83-dc814e961e89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:11:07.600821 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600770 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e64aba5-a055-4e97-8a83-dc814e961e89-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:07.600821 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600791 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:07.600821 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600806 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:07.600821 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600822 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:07.601112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.600836 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:07.602515 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.602483 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-kube-api-access-b49x4" pod "8e64aba5-a055-4e97-8a83-dc814e961e89" (UID: "8e64aba5-a055-4e97-8a83-dc814e961e89"). InnerVolumeSpecName "kube-api-access-b49x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:11:07.602623 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.602519 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-token" pod "8e64aba5-a055-4e97-8a83-dc814e961e89" (UID: "8e64aba5-a055-4e97-8a83-dc814e961e89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:11:07.602623 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.602566 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-envoy" pod "8e64aba5-a055-4e97-8a83-dc814e961e89" (UID: "8e64aba5-a055-4e97-8a83-dc814e961e89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:11:07.602623 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.602578 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-podinfo" pod "8e64aba5-a055-4e97-8a83-dc814e961e89" (UID: "8e64aba5-a055-4e97-8a83-dc814e961e89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:11:07.701582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.701549 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:07.701582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.701578 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:07.701582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.701589 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e64aba5-a055-4e97-8a83-dc814e961e89-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:07.701834 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.701602 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b49x4\" (UniqueName: \"kubernetes.io/projected/8e64aba5-a055-4e97-8a83-dc814e961e89-kube-api-access-b49x4\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:11:07.951138 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.951102 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 20:11:07.951317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:07.951159 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" podUID="b39528d6-5e81-437f-b049-79e42cc4f6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 20:11:08.370984 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:08.370942 5561 generic.go:356] "Generic (PLEG): container finished" podID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerID="b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b" exitCode=0 Nov 02 20:11:08.371480 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:08.370995 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" Nov 02 20:11:08.371480 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:08.371029 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" event={"ID":"8e64aba5-a055-4e97-8a83-dc814e961e89","Type":"ContainerDied","Data":"b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b"} Nov 02 20:11:08.371480 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:08.371088 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s" event={"ID":"8e64aba5-a055-4e97-8a83-dc814e961e89","Type":"ContainerDied","Data":"14e60311c775bac0ef0a56abf0a314457a189ef0ec220805b1d71b509efd56c7"} Nov 02 20:11:08.371480 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:08.371118 5561 scope.go:122] "RemoveContainer" containerID="b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b" Nov 02 20:11:08.384240 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:08.384212 5561 scope.go:122] "RemoveContainer" containerID="b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b" Nov 02 20:11:08.384536 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:11:08.384518 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b\": container with ID starting with b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b not found: ID does not exist" containerID="b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b" Nov 02 20:11:08.384621 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:08.384544 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b"} err="failed to get container status \"b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b\": rpc error: code = NotFound desc = could not find container \"b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b\": container with ID starting with b8317bd057d00739ab5726f442d1b9710e80e72035e7abefe4053cdc32cd254b not found: ID does not exist" Nov 02 20:11:08.574315 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:08.574281 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s"] Nov 02 20:11:08.577354 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:08.577312 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qk82s"] Nov 02 20:11:08.954916 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:08.954887 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:08.955195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:08.955176 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:08.955786 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:08.955768 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:11:09.313491 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:09.313411 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" path="/var/lib/kubelet/pods/8e64aba5-a055-4e97-8a83-dc814e961e89/volumes" Nov 02 20:11:09.483868 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:09.483828 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:10.522009 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:10.521974 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:15.048746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:15.048698 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:16.112204 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.112168 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz"] Nov 02 20:11:16.112638 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.112625 5561 cpu_manager.go:414] "containerMap: removing container" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" Nov 02 20:11:16.112638 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.112636 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" Nov 02 20:11:16.112724 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.112707 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e64aba5-a055-4e97-8a83-dc814e961e89" containerName="istio-proxy" Nov 02 20:11:16.117340 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.117318 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.166921 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.166861 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-cc64d3-openshift-default-dockercfg-dq8fc\"" Nov 02 20:11:16.194600 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.194560 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.194600 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.194600 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.194810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.194722 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c088f7da-23e3-4952-bb72-a9e91456d9a3-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.194810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.194745 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.194810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.194764 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.194961 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.194847 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.194961 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.194943 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8486\" (UniqueName: \"kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-kube-api-access-q8486\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.195047 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.195026 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.195135 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.195122 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.217134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.217092 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz"] Nov 02 20:11:16.296615 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.296571 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.296813 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.296629 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.296813 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.296656 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.296813 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.296719 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c088f7da-23e3-4952-bb72-a9e91456d9a3-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.296813 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.296747 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.296813 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.296778 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.297122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.296836 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.297122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.296871 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8486\" (UniqueName: \"kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-kube-api-access-q8486\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.297122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.296968 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.297271 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.297122 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.297271 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.297188 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.297271 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.297235 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.297271 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.297252 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.297448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.297433 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c088f7da-23e3-4952-bb72-a9e91456d9a3-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.299264 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.299241 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.299372 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.299351 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.351255 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.351220 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.351415 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.351307 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8486\" (UniqueName: \"kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-kube-api-access-q8486\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.430392 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.430357 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:16.856449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.856365 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz"] Nov 02 20:11:16.863358 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:11:16.863328 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc088f7da_23e3_4952_bb72_a9e91456d9a3.slice/crio-658191d851cc398a111f760fe1c4378197337f2693a12e23cf6f8dca8fe9f3e1 WatchSource:0}: Error finding container 658191d851cc398a111f760fe1c4378197337f2693a12e23cf6f8dca8fe9f3e1: Status 404 returned error can't find the container with id 658191d851cc398a111f760fe1c4378197337f2693a12e23cf6f8dca8fe9f3e1 Nov 02 20:11:16.865831 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.865791 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:11:16.865940 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.865865 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:11:16.865940 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:16.865935 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:11:17.426754 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:17.426710 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" event={"ID":"c088f7da-23e3-4952-bb72-a9e91456d9a3","Type":"ContainerStarted","Data":"1142ad4f3a677345bbb04cb24fbef6ab670bfcff393f9e56fa61c069e00f4568"} Nov 02 20:11:17.427201 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:17.426769 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" event={"ID":"c088f7da-23e3-4952-bb72-a9e91456d9a3","Type":"ContainerStarted","Data":"658191d851cc398a111f760fe1c4378197337f2693a12e23cf6f8dca8fe9f3e1"} Nov 02 20:11:17.439902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:17.437267 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:17.453931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:17.452215 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:11:17.453931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:17.452276 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" podUID="c088f7da-23e3-4952-bb72-a9e91456d9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 20:11:17.550313 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:17.550250 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" podStartSLOduration=1.5502301790000002 podStartE2EDuration="1.550230179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:17.54923037 +0000 UTC m=+1114.357014624" watchObservedRunningTime="2026-11-02 20:11:17.550230179 +0000 UTC m=+1114.358014396" Nov 02 20:11:18.453096 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:18.453041 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:11:18.453623 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:18.453108 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" podUID="c088f7da-23e3-4952-bb72-a9e91456d9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 20:11:19.446900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:19.445905 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:11:19.446900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:19.445978 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" podUID="c088f7da-23e3-4952-bb72-a9e91456d9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 20:11:19.656495 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:19.653417 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:20.445900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:20.443282 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:11:20.445900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:20.443357 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" podUID="c088f7da-23e3-4952-bb72-a9e91456d9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 20:11:21.446312 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:21.446271 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:11:21.446964 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:21.446928 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" podUID="c088f7da-23e3-4952-bb72-a9e91456d9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 20:11:22.432349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:22.432308 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:11:22.432667 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:22.432632 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" podUID="c088f7da-23e3-4952-bb72-a9e91456d9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 20:11:23.431592 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:23.431549 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:11:23.432061 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:23.431621 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" podUID="c088f7da-23e3-4952-bb72-a9e91456d9a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 20:11:24.162680 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:24.162643 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:24.435409 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:24.435331 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:24.435796 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:24.435734 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:24.436414 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:24.436396 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:11:28.623864 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:28.623826 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:31.915237 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:31.915200 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz"] Nov 02 20:11:31.919443 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:31.919403 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:31.972254 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:31.972220 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz"] Nov 02 20:11:31.994577 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:31.994544 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj6m2\" (UniqueName: \"kubernetes.io/projected/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-kube-api-access-wj6m2\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:31.994746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:31.994594 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-model-cache\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:31.994746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:31.994682 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-dshm\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:31.994746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:31.994732 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-home\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:31.994856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:31.994802 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tmp-dir\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:31.994856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:31.994825 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tls-certs\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.016357 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.016323 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-982b8a\"/\"distinct-2-w6-982b8a-kserve-self-signed-certs\"" Nov 02 20:11:32.053797 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.053739 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:32.096036 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.096000 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj6m2\" (UniqueName: \"kubernetes.io/projected/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-kube-api-access-wj6m2\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.096234 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.096045 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-model-cache\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.096234 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.096078 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-dshm\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.096234 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.096112 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-home\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.096234 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.096133 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tmp-dir\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.096234 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.096148 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tls-certs\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.096621 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.096591 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-model-cache\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.096800 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.096617 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-home\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.096918 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.096694 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tmp-dir\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.099153 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.099130 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-dshm\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.099316 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.099298 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tls-certs\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.150693 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.150666 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj6m2\" (UniqueName: \"kubernetes.io/projected/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-kube-api-access-wj6m2\") pod \"distinct-2-w6-982b8a-kserve-5cd56486-qmbsz\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.236594 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.236502 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:32.626517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:32.626436 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz"] Nov 02 20:11:32.634008 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:11:32.633982 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef8d55e1_2ef6_4a45_b4f2_98aefb12b870.slice/crio-66dd6647f58caad21198485a01c4dde716f7a1188c098b465917ca0109141a18 WatchSource:0}: Error finding container 66dd6647f58caad21198485a01c4dde716f7a1188c098b465917ca0109141a18: Status 404 returned error can't find the container with id 66dd6647f58caad21198485a01c4dde716f7a1188c098b465917ca0109141a18 Nov 02 20:11:33.250021 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:33.249986 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:33.541901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:33.541807 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" event={"ID":"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870","Type":"ContainerStarted","Data":"48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75"} Nov 02 20:11:33.541901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:33.541852 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" event={"ID":"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870","Type":"ContainerStarted","Data":"66dd6647f58caad21198485a01c4dde716f7a1188c098b465917ca0109141a18"} Nov 02 20:11:33.650923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:33.650854 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podStartSLOduration=2.650839183 podStartE2EDuration="2.650839183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:33.649940883 +0000 UTC m=+1130.457725123" watchObservedRunningTime="2026-11-02 20:11:33.650839183 +0000 UTC m=+1130.458623398" Nov 02 20:11:33.784091 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:11:33.784048 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:11:34.237597 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:34.237566 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:37.826183 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:37.826150 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:42.237418 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:42.237378 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:42.248110 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:42.248065 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:42.250529 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:42.250511 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:42.599675 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:42.599585 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:11:46.753609 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:46.753570 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:47.143641 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:47.143554 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:47.433746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:47.433707 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:51.390410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:51.390376 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:11:55.952571 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:11:55.952523 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:00.585799 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:00.585757 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:01.409327 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.409294 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z"] Nov 02 20:12:01.412950 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.412930 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.537521 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.537477 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.537719 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.537602 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.537719 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.537637 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.537719 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.537711 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.537966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.537808 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd7b76c6-3cb5-48f1-8347-2fe977212038-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.537966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.537900 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.537966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.537937 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.538107 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.538009 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.538107 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.538056 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29fq5\" (UniqueName: \"kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-kube-api-access-29fq5\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.604672 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.604638 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z"] Nov 02 20:12:01.639204 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639165 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.639395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639212 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.639395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639254 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.639395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639295 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd7b76c6-3cb5-48f1-8347-2fe977212038-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.639395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639347 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.639395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639379 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.639670 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639440 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.639670 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639502 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29fq5\" (UniqueName: \"kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-kube-api-access-29fq5\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.639670 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639555 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.639836 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639700 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.639912 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639861 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.639976 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639931 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.640029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.639978 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.640253 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.640237 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd7b76c6-3cb5-48f1-8347-2fe977212038-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.641736 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.641711 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.641966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.641947 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.694544 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.694471 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29fq5\" (UniqueName: \"kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-kube-api-access-29fq5\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.694698 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.694562 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.725587 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.725548 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt"] Nov 02 20:12:01.725777 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.725617 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:01.730150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.729854 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.784344 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.784079 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-982b8a-openshift-default-dockercfg-lsv7m\"" Nov 02 20:12:01.786743 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.786708 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt"] Nov 02 20:12:01.850445 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.842443 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.850445 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.842588 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.850445 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.842740 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.850445 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.842808 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.850445 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.842846 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgw6r\" (UniqueName: \"kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-kube-api-access-xgw6r\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.850445 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.842933 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.850445 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.842969 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.850445 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.843025 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.850445 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.843077 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.945547 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.945458 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.945547 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.945518 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.945856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.945547 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgw6r\" (UniqueName: \"kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-kube-api-access-xgw6r\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.945856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.945588 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.945856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.945617 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.945856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.945663 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.945856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.945706 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.945856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.945797 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.945856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.945843 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.946301 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.946150 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.946301 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.946175 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.946301 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.946288 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.946422 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.946311 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.946422 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.946394 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.948316 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.948294 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:01.948496 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:01.948475 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:02.000306 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.000274 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:02.000461 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.000335 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgw6r\" (UniqueName: \"kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-kube-api-access-xgw6r\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:02.056441 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.056406 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:02.152941 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.152907 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z"] Nov 02 20:12:02.161189 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:12:02.161153 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7b76c6_3cb5_48f1_8347_2fe977212038.slice/crio-381b42f78e866c9636b3f621c00081c8762b2b6eeae71a7cb9185ce133daef41 WatchSource:0}: Error finding container 381b42f78e866c9636b3f621c00081c8762b2b6eeae71a7cb9185ce133daef41: Status 404 returned error can't find the container with id 381b42f78e866c9636b3f621c00081c8762b2b6eeae71a7cb9185ce133daef41 Nov 02 20:12:02.164347 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.164304 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:12:02.164447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.164384 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:12:02.164447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.164421 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:12:02.428197 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.428158 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt"] Nov 02 20:12:02.436258 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:12:02.436229 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e2f75e_18ea_4b40_bada_758f81aa07b0.slice/crio-5d5494c9c0bd0eaf8cc329a71cf25579619e900edb32606bf4f8262273e0731f WatchSource:0}: Error finding container 5d5494c9c0bd0eaf8cc329a71cf25579619e900edb32606bf4f8262273e0731f: Status 404 returned error can't find the container with id 5d5494c9c0bd0eaf8cc329a71cf25579619e900edb32606bf4f8262273e0731f Nov 02 20:12:02.439130 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.439090 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:12:02.439213 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.439192 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:12:02.439261 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.439244 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:12:02.704314 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.704215 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" event={"ID":"e3e2f75e-18ea-4b40-bada-758f81aa07b0","Type":"ContainerStarted","Data":"5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9"} Nov 02 20:12:02.704314 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.704281 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" event={"ID":"e3e2f75e-18ea-4b40-bada-758f81aa07b0","Type":"ContainerStarted","Data":"5d5494c9c0bd0eaf8cc329a71cf25579619e900edb32606bf4f8262273e0731f"} Nov 02 20:12:02.706423 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.706391 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" event={"ID":"bd7b76c6-3cb5-48f1-8347-2fe977212038","Type":"ContainerStarted","Data":"f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35"} Nov 02 20:12:02.706548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.706441 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" event={"ID":"bd7b76c6-3cb5-48f1-8347-2fe977212038","Type":"ContainerStarted","Data":"381b42f78e866c9636b3f621c00081c8762b2b6eeae71a7cb9185ce133daef41"} Nov 02 20:12:02.726678 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.726653 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:02.817337 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.817280 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" podStartSLOduration=1.817266247 podStartE2EDuration="1.817266247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:02.815225968 +0000 UTC m=+1159.623010185" watchObservedRunningTime="2026-11-02 20:12:02.817266247 +0000 UTC m=+1159.625050454" Nov 02 20:12:02.931897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:02.928610 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" podStartSLOduration=1.928591151 podStartE2EDuration="1.928591151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:02.92599709 +0000 UTC m=+1159.733781309" watchObservedRunningTime="2026-11-02 20:12:02.928591151 +0000 UTC m=+1159.736375368" Nov 02 20:12:03.062960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:03.062674 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:03.062960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:03.062818 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 20:12:03.062960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:03.062863 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" podUID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 20:12:03.727162 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:03.727130 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 20:12:03.727620 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:03.727597 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" podUID="bd7b76c6-3cb5-48f1-8347-2fe977212038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 20:12:03.807648 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:12:03.807607 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:12:04.067957 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:04.064260 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 20:12:04.067957 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:04.064318 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" podUID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 20:12:04.726160 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:04.726127 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 20:12:04.726345 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:04.726177 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" podUID="bd7b76c6-3cb5-48f1-8347-2fe977212038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 20:12:05.057958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:05.057244 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 20:12:05.057958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:05.057309 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" podUID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 20:12:05.225800 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:05.225753 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:05.760173 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:05.760133 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 20:12:05.778043 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:05.777989 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" podUID="bd7b76c6-3cb5-48f1-8347-2fe977212038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 20:12:06.057958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:06.057285 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 20:12:06.057958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:06.057366 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" podUID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 20:12:06.726328 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:06.726292 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 20:12:06.726517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:06.726343 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" podUID="bd7b76c6-3cb5-48f1-8347-2fe977212038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 20:12:07.057955 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:07.057252 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 20:12:07.057955 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:07.057329 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" podUID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 20:12:07.726206 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:07.726176 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 20:12:07.726627 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:07.726603 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" podUID="bd7b76c6-3cb5-48f1-8347-2fe977212038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 20:12:08.057958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:08.057246 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 20:12:08.057958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:08.057308 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" podUID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 20:12:08.726437 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:08.726393 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 20:12:08.726828 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:08.726465 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" podUID="bd7b76c6-3cb5-48f1-8347-2fe977212038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 20:12:09.057586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:09.057483 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 20:12:09.057586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:09.057553 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" podUID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 20:12:09.732277 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:09.732242 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:09.732769 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:09.732605 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:09.733320 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:09.733298 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:12:09.965189 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:09.965142 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:10.063179 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:10.063093 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:10.063553 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:10.063521 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:10.064447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:10.064427 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:12:13.343850 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:13.343807 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:14.562258 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:14.562210 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:19.178883 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:19.178825 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:23.609651 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:23.609602 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:24.776024 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.775982 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c"] Nov 02 20:12:24.783270 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.783241 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:24.833659 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.833625 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c"] Nov 02 20:12:24.877779 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.877744 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"default-dockercfg-94w94\"" Nov 02 20:12:24.877779 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.877771 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"distinct-2-w3-2052e3-kserve-self-signed-certs\"" Nov 02 20:12:24.880714 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.880695 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"kube-root-ca.crt\"" Nov 02 20:12:24.880803 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.880730 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"openshift-service-ca.crt\"" Nov 02 20:12:24.910072 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.910039 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:24.959170 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.959132 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-tmp-dir\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:24.959349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.959178 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcbd4\" (UniqueName: \"kubernetes.io/projected/3751f81c-d14d-420c-8cc7-273e186114a1-kube-api-access-wcbd4\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:24.959349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.959247 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-model-cache\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:24.959349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.959337 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-dshm\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:24.959502 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.959468 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3751f81c-d14d-420c-8cc7-273e186114a1-tls-certs\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:24.959553 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:24.959519 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-home\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.060995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.060896 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-dshm\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.060995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.060982 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3751f81c-d14d-420c-8cc7-273e186114a1-tls-certs\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.061210 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.061003 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-home\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.061210 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.061095 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-tmp-dir\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.061210 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.061117 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcbd4\" (UniqueName: \"kubernetes.io/projected/3751f81c-d14d-420c-8cc7-273e186114a1-kube-api-access-wcbd4\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.061210 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.061141 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-model-cache\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.061551 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.061526 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-tmp-dir\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.061551 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.061539 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-model-cache\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.061703 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.061644 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-home\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.063937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.063908 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-dshm\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.064108 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.064086 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3751f81c-d14d-420c-8cc7-273e186114a1-tls-certs\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.115608 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.115572 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcbd4\" (UniqueName: \"kubernetes.io/projected/3751f81c-d14d-420c-8cc7-273e186114a1-kube-api-access-wcbd4\") pod \"distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.397913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.397795 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:25.768272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.768241 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c"] Nov 02 20:12:25.775367 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:12:25.775336 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3751f81c_d14d_420c_8cc7_273e186114a1.slice/crio-416816ae4e46a2e89fb36fdcef5f40b6f37c2c6ae413498bca984d9c09d95800 WatchSource:0}: Error finding container 416816ae4e46a2e89fb36fdcef5f40b6f37c2c6ae413498bca984d9c09d95800: Status 404 returned error can't find the container with id 416816ae4e46a2e89fb36fdcef5f40b6f37c2c6ae413498bca984d9c09d95800 Nov 02 20:12:25.887109 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.887076 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" event={"ID":"3751f81c-d14d-420c-8cc7-273e186114a1","Type":"ContainerStarted","Data":"ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee"} Nov 02 20:12:25.887496 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.887120 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" event={"ID":"3751f81c-d14d-420c-8cc7-273e186114a1","Type":"ContainerStarted","Data":"416816ae4e46a2e89fb36fdcef5f40b6f37c2c6ae413498bca984d9c09d95800"} Nov 02 20:12:25.995668 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:25.995615 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podStartSLOduration=1.9956014180000001 podStartE2EDuration="1.995601418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:25.994373796 +0000 UTC m=+1182.802158012" watchObservedRunningTime="2026-11-02 20:12:25.995601418 +0000 UTC m=+1182.803385664" Nov 02 20:12:26.398517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:26.398488 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:28.158230 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:28.158192 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:32.203344 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.203305 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r"] Nov 02 20:12:32.207072 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.207049 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.238476 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.238425 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-dshm\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.238640 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.238492 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-home\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.238708 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.238632 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tls-certs\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.238763 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.238721 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbwdb\" (UniqueName: \"kubernetes.io/projected/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-kube-api-access-xbwdb\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.238848 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.238833 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tmp-dir\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.238935 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.238915 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-model-cache\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.304841 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.304813 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ae8e6e\"/\"distinct-2-w5-ae8e6e-kserve-self-signed-certs\"" Nov 02 20:12:32.306762 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.306734 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r"] Nov 02 20:12:32.338688 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.338651 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:32.339742 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.339709 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tmp-dir\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.339909 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.339774 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-model-cache\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.339969 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.339919 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-dshm\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.340020 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.340002 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-home\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.340620 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.340192 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tls-certs\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.340620 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.340286 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbwdb\" (UniqueName: \"kubernetes.io/projected/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-kube-api-access-xbwdb\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.340620 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.340370 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-model-cache\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.340620 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.340440 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-home\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.340620 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.340466 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tmp-dir\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.342776 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.342749 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-dshm\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.343074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.343052 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tls-certs\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.394382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.394344 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbwdb\" (UniqueName: \"kubernetes.io/projected/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-kube-api-access-xbwdb\") pod \"distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.520622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.520537 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:32.682848 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.682818 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:32.900327 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.900251 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r"] Nov 02 20:12:32.907994 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:12:32.907966 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51e6bd1e_cd4e_45a4_a9da_9003f53f359e.slice/crio-e2def228f2c0b2f32988fb34eed5c95fac364a5811debde8698e155b14a7c996 WatchSource:0}: Error finding container e2def228f2c0b2f32988fb34eed5c95fac364a5811debde8698e155b14a7c996: Status 404 returned error can't find the container with id e2def228f2c0b2f32988fb34eed5c95fac364a5811debde8698e155b14a7c996 Nov 02 20:12:32.928710 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:32.928682 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" event={"ID":"51e6bd1e-cd4e-45a4-a9da-9003f53f359e","Type":"ContainerStarted","Data":"e2def228f2c0b2f32988fb34eed5c95fac364a5811debde8698e155b14a7c996"} Nov 02 20:12:33.780175 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:12:33.780138 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:12:33.936065 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:33.936026 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" event={"ID":"51e6bd1e-cd4e-45a4-a9da-9003f53f359e","Type":"ContainerStarted","Data":"7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47"} Nov 02 20:12:34.046664 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:34.046552 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podStartSLOduration=2.046539306 podStartE2EDuration="2.046539306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:34.044246615 +0000 UTC m=+1190.852030859" watchObservedRunningTime="2026-11-02 20:12:34.046539306 +0000 UTC m=+1190.854323522" Nov 02 20:12:34.521671 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:34.521636 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:35.398651 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:35.398617 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:35.411257 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:35.411232 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:35.957492 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:35.957464 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:12:37.257062 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:37.257025 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:41.776518 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:41.776483 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:42.521779 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:42.521731 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:42.540150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:42.540118 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:42.995351 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:42.995325 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:12:46.260402 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:46.260365 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:46.551405 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:46.551301 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:47.484747 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:47.484715 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:51.074645 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:51.074612 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:12:55.465439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:12:55.465380 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:00.005737 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:00.005647 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:00.994241 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:00.991168 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:01.195901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:01.191455 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:01.706901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:01.704026 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:03.781914 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:13:03.781753 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:13:04.862533 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:04.862496 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:09.383348 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:09.383307 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:14.167934 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:14.167893 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:16.646741 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.646707 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc"] Nov 02 20:13:16.651016 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.650995 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.736741 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.736706 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzlkt\" (UniqueName: \"kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-kube-api-access-kzlkt\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.736741 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.736742 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.736990 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.736758 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df0c17b8-8501-45ac-8667-4a24b770b638-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.736990 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.736947 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df0c17b8-8501-45ac-8667-4a24b770b638-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.737101 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.736992 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.737157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.737105 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.737259 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.737242 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.737374 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.737347 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.737429 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.737414 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.836221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.836191 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc"] Nov 02 20:13:16.838399 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.838374 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.838539 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.838429 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.838539 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.838468 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.838539 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.838495 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzlkt\" (UniqueName: \"kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-kube-api-access-kzlkt\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.838539 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.838518 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.838766 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.838543 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df0c17b8-8501-45ac-8667-4a24b770b638-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.838766 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.838589 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df0c17b8-8501-45ac-8667-4a24b770b638-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.838766 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.838613 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.838766 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.838665 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.839028 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.839001 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.839190 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.839167 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.839190 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.839120 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.839363 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.839341 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df0c17b8-8501-45ac-8667-4a24b770b638-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.839408 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.839386 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.841749 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.841718 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.841959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.841939 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df0c17b8-8501-45ac-8667-4a24b770b638-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.893110 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.893076 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.893253 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.893182 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzlkt\" (UniqueName: \"kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-kube-api-access-kzlkt\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.967258 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.967224 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:16.980655 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.980618 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt"] Nov 02 20:13:16.984489 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:16.984465 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.047964 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.047931 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt"] Nov 02 20:13:17.056628 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.056534 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.056628 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.056577 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.056830 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.056679 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.056830 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.056751 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.056830 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.056822 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrz82\" (UniqueName: \"kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-kube-api-access-wrz82\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.057014 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.056861 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.057014 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.056985 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.057167 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.057123 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.057167 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.057160 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.157783 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.157746 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrz82\" (UniqueName: \"kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-kube-api-access-wrz82\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158020 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.157800 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158020 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.157856 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158020 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.157999 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158193 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.158031 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158193 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.158099 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158193 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.158127 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158193 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.158176 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158450 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.158230 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158450 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.158408 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158819 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.158768 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158819 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.158802 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158993 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.158823 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.158993 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.158868 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.160769 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.160741 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.160862 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.160852 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.212358 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.212320 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.212542 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.212381 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrz82\" (UniqueName: \"kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-kube-api-access-wrz82\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.291093 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.291020 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p"] Nov 02 20:13:17.295937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.295916 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.305734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.305712 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:17.354104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.348373 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p"] Nov 02 20:13:17.360058 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.360027 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb9d013-92c8-426b-9127-eaac2797978f-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.360058 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.360060 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb9d013-92c8-426b-9127-eaac2797978f-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.360497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.360471 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.360599 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.360584 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.360761 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.360738 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.360831 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.360806 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.360980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.360958 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.361147 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.361129 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.361220 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.361202 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pqdd\" (UniqueName: \"kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-kube-api-access-4pqdd\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.401418 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.401384 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc"] Nov 02 20:13:17.409727 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:13:17.409686 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0c17b8_8501_45ac_8667_4a24b770b638.slice/crio-6dd5c93320940dca0e2e41dfcdc094af6fa31bfa06d6776f0a2dbc9e896c2cb3 WatchSource:0}: Error finding container 6dd5c93320940dca0e2e41dfcdc094af6fa31bfa06d6776f0a2dbc9e896c2cb3: Status 404 returned error can't find the container with id 6dd5c93320940dca0e2e41dfcdc094af6fa31bfa06d6776f0a2dbc9e896c2cb3 Nov 02 20:13:17.412702 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.412664 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:17.412798 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.412737 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:17.412798 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.412771 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:17.462007 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.461980 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.462143 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.462047 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.462143 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.462074 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.462143 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.462132 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.462332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.462217 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.462601 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.462549 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.462601 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.462399 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.462601 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.462583 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pqdd\" (UniqueName: \"kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-kube-api-access-4pqdd\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.462926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.462620 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.462926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.462643 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb9d013-92c8-426b-9127-eaac2797978f-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.462926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.462693 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb9d013-92c8-426b-9127-eaac2797978f-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.462926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.462748 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.463159 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.463112 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.463463 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.463436 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb9d013-92c8-426b-9127-eaac2797978f-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.465020 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.464996 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.465100 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.465022 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb9d013-92c8-426b-9127-eaac2797978f-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.516196 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.516163 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.516330 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.516221 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pqdd\" (UniqueName: \"kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-kube-api-access-4pqdd\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.609983 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.609907 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:17.686618 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.686586 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt"] Nov 02 20:13:17.694866 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:13:17.694831 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1780ec_55b0_41e8_bd7d_ba8f7d369f66.slice/crio-75cf158d1461d1e0be3d33ae00ea5ec6b56341dab5e4e98aa4ecfbc419455a19 WatchSource:0}: Error finding container 75cf158d1461d1e0be3d33ae00ea5ec6b56341dab5e4e98aa4ecfbc419455a19: Status 404 returned error can't find the container with id 75cf158d1461d1e0be3d33ae00ea5ec6b56341dab5e4e98aa4ecfbc419455a19 Nov 02 20:13:17.700856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.700823 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:17.700977 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.700916 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:17.700977 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.700955 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:17.900160 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.900085 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc"] Nov 02 20:13:17.905055 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.905032 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:17.925272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.925244 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt"] Nov 02 20:13:17.929302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.929280 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:17.957114 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.957076 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc"] Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973406 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973453 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973484 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973511 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973536 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fffxs\" (UniqueName: \"kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-kube-api-access-fffxs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973578 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973611 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973650 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973680 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973705 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973752 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9814eaca-0ca2-4925-ae97-ea136e05daff-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973841 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztkrw\" (UniqueName: \"kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-kube-api-access-ztkrw\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973897 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973968 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.973993 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.974171 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.974196 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:17.974513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.974245 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:17.994762 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:17.994726 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p"] Nov 02 20:13:18.021280 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.021247 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt"] Nov 02 20:13:18.075962 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.075927 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.077115 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.077088 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.077290 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.077274 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.077433 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.077418 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9814eaca-0ca2-4925-ae97-ea136e05daff-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.077559 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.077545 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztkrw\" (UniqueName: \"kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-kube-api-access-ztkrw\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.077677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.077655 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.077803 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.077768 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.077926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.077842 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:18.078046 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.078027 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.078159 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.078141 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.078298 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.078046 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:18.078430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.078396 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:18.078499 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.078470 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.078545 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.078528 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.078650 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.078624 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9814eaca-0ca2-4925-ae97-ea136e05daff-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.078721 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.078696 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.078852 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.078729 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.078852 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.078777 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.078852 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.078816 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.079104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.078988 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.079104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.079021 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.079104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.079060 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.079104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.079091 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fffxs\" (UniqueName: \"kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-kube-api-access-fffxs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.079310 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.079131 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.079310 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.079143 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.079310 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.079152 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.079310 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.079206 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.079715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.079407 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.079715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.079566 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.080523 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.080499 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.080759 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.080738 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.080759 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.080746 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.089390 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.089363 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.089493 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.089416 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.089493 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.089448 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.166598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.165850 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.169139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.169096 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztkrw\" (UniqueName: \"kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-kube-api-access-ztkrw\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.169520 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.169491 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fffxs\" (UniqueName: \"kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-kube-api-access-fffxs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.170132 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.170102 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.215357 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.215320 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" event={"ID":"df0c17b8-8501-45ac-8667-4a24b770b638","Type":"ContainerStarted","Data":"5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc"} Nov 02 20:13:18.215523 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.215418 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" event={"ID":"df0c17b8-8501-45ac-8667-4a24b770b638","Type":"ContainerStarted","Data":"6dd5c93320940dca0e2e41dfcdc094af6fa31bfa06d6776f0a2dbc9e896c2cb3"} Nov 02 20:13:18.218915 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.218851 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" event={"ID":"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66","Type":"ContainerStarted","Data":"d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866"} Nov 02 20:13:18.219045 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.218939 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" event={"ID":"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66","Type":"ContainerStarted","Data":"75cf158d1461d1e0be3d33ae00ea5ec6b56341dab5e4e98aa4ecfbc419455a19"} Nov 02 20:13:18.220693 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.220593 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:18.222462 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.222437 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" event={"ID":"cdb9d013-92c8-426b-9127-eaac2797978f","Type":"ContainerStarted","Data":"709a4646e23453c1ea631b0dfbfbbe5ac188355a47adaea53a319cb0b1bc07f0"} Nov 02 20:13:18.222553 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.222486 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" event={"ID":"cdb9d013-92c8-426b-9127-eaac2797978f","Type":"ContainerStarted","Data":"fe8d2c609c5fceb094c88bab6600af09cbabb9068bd45fbf4816a0de93f88bcd"} Nov 02 20:13:18.262551 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.262508 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:18.264388 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.264335 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w"] Nov 02 20:13:18.271207 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.271179 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.281740 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.281713 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.281955 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.281937 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.282085 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.282069 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnkkp\" (UniqueName: \"kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-kube-api-access-wnkkp\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.282273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.282260 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.282437 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.282422 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.282528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.282515 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.282614 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.282601 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8e9e7a-7797-4186-916f-15c6e80b7098-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.282826 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.282812 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.283999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.283977 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.306331 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.306304 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:18.314349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.314326 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w"] Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.394813 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.394897 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.396156 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.396242 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.396285 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnkkp\" (UniqueName: \"kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-kube-api-access-wnkkp\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.396370 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.396474 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.396505 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.396533 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8e9e7a-7797-4186-916f-15c6e80b7098-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.396597 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.396644 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.397302 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.397737 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8e9e7a-7797-4186-916f-15c6e80b7098-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398795 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.398321 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398795 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.398555 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.398795 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.398748 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.401765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.401740 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.401986 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.401965 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.477954 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.464448 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.511283 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.505743 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnkkp\" (UniqueName: \"kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-kube-api-access-wnkkp\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.511283 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.508268 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podStartSLOduration=2.5082510620000003 podStartE2EDuration="2.508251062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:18.39114179 +0000 UTC m=+1235.198926007" watchObservedRunningTime="2026-11-02 20:13:18.508251062 +0000 UTC m=+1235.316035278" Nov 02 20:13:18.511283 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.509035 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" podStartSLOduration=1.509019408 podStartE2EDuration="1.509019408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:18.507598318 +0000 UTC m=+1235.315382559" watchObservedRunningTime="2026-11-02 20:13:18.509019408 +0000 UTC m=+1235.316803626" Nov 02 20:13:18.610826 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.610791 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:18.620086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.619757 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:18.806972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.765530 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:18.808665 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.808004 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podStartSLOduration=2.807977386 podStartE2EDuration="2.807977386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:18.765373052 +0000 UTC m=+1235.573157273" watchObservedRunningTime="2026-11-02 20:13:18.807977386 +0000 UTC m=+1235.615761603" Nov 02 20:13:18.909317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.903960 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc"] Nov 02 20:13:18.915831 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.915749 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt"] Nov 02 20:13:18.918263 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:13:18.918234 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a341b73_b2fd_46f6_b81a_8b7bf2f06355.slice/crio-a6604d799f7d1a117c2022207f01e03997651c5fd913f02bb2e31b376e7fe42c WatchSource:0}: Error finding container a6604d799f7d1a117c2022207f01e03997651c5fd913f02bb2e31b376e7fe42c: Status 404 returned error can't find the container with id a6604d799f7d1a117c2022207f01e03997651c5fd913f02bb2e31b376e7fe42c Nov 02 20:13:18.920280 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:13:18.920241 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9814eaca_0ca2_4925_ae97_ea136e05daff.slice/crio-014416cd32ad27dc292261623b3ecb7c3c3e630014a9ca5ca8e47b55245c2b24 WatchSource:0}: Error finding container 014416cd32ad27dc292261623b3ecb7c3c3e630014a9ca5ca8e47b55245c2b24: Status 404 returned error can't find the container with id 014416cd32ad27dc292261623b3ecb7c3c3e630014a9ca5ca8e47b55245c2b24 Nov 02 20:13:18.921845 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.921807 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:18.922007 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.921939 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:18.922007 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.921988 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:18.926125 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.925449 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:18.926125 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.925526 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:18.926125 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.925568 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:18.978973 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.977811 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:18.978973 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.977948 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 20:13:18.978973 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:18.977994 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 20:13:19.189294 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.189184 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w"] Nov 02 20:13:19.219248 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.219201 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:19.219361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.219298 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:19.219361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.219347 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:19.287286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.259482 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" event={"ID":"7a341b73-b2fd-46f6-b81a-8b7bf2f06355","Type":"ContainerStarted","Data":"b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309"} Nov 02 20:13:19.287286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.259552 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" event={"ID":"7a341b73-b2fd-46f6-b81a-8b7bf2f06355","Type":"ContainerStarted","Data":"a6604d799f7d1a117c2022207f01e03997651c5fd913f02bb2e31b376e7fe42c"} Nov 02 20:13:19.287286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.285491 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:19.320046 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.316178 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 20:13:19.320046 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.316235 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 20:13:19.504983 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.504945 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" event={"ID":"9814eaca-0ca2-4925-ae97-ea136e05daff","Type":"ContainerStarted","Data":"014416cd32ad27dc292261623b3ecb7c3c3e630014a9ca5ca8e47b55245c2b24"} Nov 02 20:13:19.505203 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.505180 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" event={"ID":"da8e9e7a-7797-4186-916f-15c6e80b7098","Type":"ContainerStarted","Data":"d537672b74f114e65616d21ae75d24b0213782d551d6fd9668a7d55cd1114e95"} Nov 02 20:13:19.640065 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.640027 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 20:13:19.642221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.642184 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 20:13:19.973900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.973726 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 20:13:19.973900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:19.973799 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 20:13:20.268134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.268045 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:13:20.268373 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.268342 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 20:13:20.309901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.308242 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 20:13:20.309901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.308303 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 20:13:20.428550 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.428507 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" event={"ID":"9814eaca-0ca2-4925-ae97-ea136e05daff","Type":"ContainerStarted","Data":"1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438"} Nov 02 20:13:20.464901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.464045 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" event={"ID":"da8e9e7a-7797-4186-916f-15c6e80b7098","Type":"ContainerStarted","Data":"1688b9bd2353fbb1651199c4c9c6b602a622fc84e441bbbfa68d30a150bfee36"} Nov 02 20:13:20.470913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.468989 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:20.564255 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.564129 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podStartSLOduration=3.564106767 podStartE2EDuration="3.564106767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:19.386137345 +0000 UTC m=+1236.193921589" watchObservedRunningTime="2026-11-02 20:13:20.564106767 +0000 UTC m=+1237.371890984" Nov 02 20:13:20.598033 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.597982 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:20.598646 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.598617 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 02 20:13:20.613294 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.613257 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 20:13:20.614570 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.614522 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 20:13:20.625093 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.625044 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:20.625695 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.625671 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:20.627912 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.627889 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:13:20.630970 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.630939 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 20:13:20.656025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.655973 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:20.698895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.676304 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podStartSLOduration=3.676283158 podStartE2EDuration="3.676283158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:20.563374559 +0000 UTC m=+1237.371158775" watchObservedRunningTime="2026-11-02 20:13:20.676283158 +0000 UTC m=+1237.484067378" Nov 02 20:13:20.702107 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.702048 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podStartSLOduration=2.702027635 podStartE2EDuration="2.702027635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:20.675353666 +0000 UTC m=+1237.483137892" watchObservedRunningTime="2026-11-02 20:13:20.702027635 +0000 UTC m=+1237.509811853" Nov 02 20:13:20.760995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.749976 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:20.875556 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.875448 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:20.945900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.940686 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:20.957009 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.956957 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded" Nov 02 20:13:20.957362 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.957337 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:20.972938 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.972897 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 20:13:20.973221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.973192 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 20:13:21.000912 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:20.999955 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.004620 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.004575 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.215781 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.214984 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.226909 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.226593 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:21.226909 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.226752 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:13:21.226909 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.226797 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 20:13:21.281359 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.281317 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:13:21.281639 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.281611 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 20:13:21.304580 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.302993 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.309268 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.309226 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 20:13:21.309428 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.309293 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 20:13:21.472631 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.472533 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.476899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.473339 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:13:21.626700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.621331 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 20:13:21.626700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.621394 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 20:13:21.626700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.621677 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:13:21.626700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.621704 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 20:13:21.626700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.621773 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.626700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.622065 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.626700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.622345 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:13:21.626700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.622400 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:13:21.632186 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.630960 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.632186 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.631336 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:13:21.876952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.845498 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 20:13:21.979418 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.850966 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.979705 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.851009 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.981366 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.981335 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.981922 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.981900 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 20:13:21.982211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.982184 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 20:13:21.982969 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.982933 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:22.020901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.985908 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:13:22.020901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.985971 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:13:22.020901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.986011 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:13:22.020901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.876140 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:22.020901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:21.988262 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:13:22.036900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.032974 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 02 20:13:22.052127 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.052095 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:13:22.155651 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.064767 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:22.155651 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.065167 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:13:22.155651 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.081911 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 20:13:22.155651 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.082292 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:13:22.155651 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.107967 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:13:22.313915 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.303278 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:22.487534 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.487184 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 20:13:22.513475 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.432416 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:22.513721 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.431035 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 20:13:22.513869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.513839 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 20:13:22.538993 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.491606 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:13:22.618333 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.431241 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:13:22.618333 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.545979 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 20:13:22.618333 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.431621 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:13:22.618333 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.546052 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 20:13:22.628899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.620976 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 20:13:22.629985 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.629948 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 20:13:22.630161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.622936 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:22.631338 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.631311 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:22.635897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.621954 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:22.638600 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.638576 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:13:22.638757 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:22.638737 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 20:13:23.043901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.043029 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 20:13:23.113977 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.113644 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.137278 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.123464 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.137278 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.123801 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.137278 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.131031 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.137278 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.131220 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 20:13:23.160901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.158011 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.165924 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.165290 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 20:13:23.165924 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.165352 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 20:13:23.323289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.303524 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:13:23.323289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.303602 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 20:13:23.323289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.303742 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:13:23.323289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.303788 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 20:13:23.323289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.315953 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 20:13:23.323289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.316261 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 20:13:23.323289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.316298 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 20:13:23.323289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.316343 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.515923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.508060 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.515923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.514236 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.620810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.611390 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 20:13:23.620810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.611468 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 20:13:23.620810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.620647 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:13:23.620810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.620697 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 20:13:23.664238 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.638979 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.664238 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.639846 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.664238 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.640376 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.664238 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.640657 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 02 20:13:23.772544 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:23.772488 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.062932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.054703 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.075945 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.070001 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 20:13:24.075945 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.070062 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 20:13:24.136078 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.136014 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 20:13:24.136847 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.136826 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:13:24.169518 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.146499 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 20:13:24.169518 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.146860 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 20:13:24.172007 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.171968 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.179928 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.172160 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 02 20:13:24.179928 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.172224 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.323975 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.304756 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:13:24.408027 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.407981 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 20:13:24.438002 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.330453 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.461901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.309360 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:13:24.463983 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.463944 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 20:13:24.464170 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.462320 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:13:24.485898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.329742 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 20:13:24.486185 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.486153 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 20:13:24.613958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.588269 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 02 20:13:24.692900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.672823 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.700265 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.700218 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 20:13:24.721904 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.712468 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Nov 02 20:13:24.742902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.723094 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.870981 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.778711 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:25.061093 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.778920 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:13:25.061365 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.061333 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 20:13:25.061525 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.980607 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.061654 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.878719 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 20:13:25.061773 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.061751 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 20:13:25.061931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:24.980521 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 20:13:25.062060 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.062041 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 20:13:25.062405 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.062375 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.161955 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.142139 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 20:13:25.166337 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.155108 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.166807 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.155196 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 20:13:25.189899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.186087 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.339923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.332985 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.339923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.333469 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.339923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.333949 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:13:25.339923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.333997 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 20:13:25.339923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.334400 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 20:13:25.339923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.334450 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 20:13:25.339923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.334524 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:13:25.339923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.334574 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 20:13:25.454954 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.438553 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 20:13:25.744971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.732955 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.744971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.733507 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.744971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.735798 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:13:25.744971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.735836 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 20:13:25.831898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.748662 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 20:13:25.846505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.846447 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 20:13:25.846763 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.758755 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.847223 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.758711 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.999978 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.996982 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 02 20:13:25.999978 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.999394 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 20:13:25.999978 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:25.999443 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 20:13:26.019902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.016247 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Nov 02 20:13:26.080952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.069689 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:26.309423 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.309317 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:26.327019 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.326971 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 20:13:26.327607 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.327576 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 20:13:26.412901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.409337 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:26.434941 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.432973 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 02 20:13:26.435385 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.433320 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 20:13:26.435528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.433354 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Nov 02 20:13:26.452024 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.451986 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 20:13:26.522941 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.493489 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:26.523345 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.453484 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:13:26.523525 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.523484 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 20:13:26.523775 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.458005 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:13:26.523950 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.523913 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 20:13:26.524123 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.524093 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 20:13:26.645248 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.645145 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 20:13:26.662765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.662726 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 20:13:26.663348 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.646373 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:13:26.683847 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.683802 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 20:13:26.742899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.736714 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:26.742899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.736829 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:26.742899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.737189 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Nov 02 20:13:26.852900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.852129 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:26.852900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.852521 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 20:13:26.969900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.968245 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 20:13:26.969900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.968306 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 20:13:26.998009 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:26.997955 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:27.041025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.040974 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:27.046098 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.046056 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded" Nov 02 20:13:27.120436 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.120389 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:27.222962 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.222291 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:13:27.222962 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.222359 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 20:13:27.268308 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.268269 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:13:27.268598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.268570 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 20:13:27.336982 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.336916 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:27.386380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.343968 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 20:13:27.386767 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.344042 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 20:13:27.391326 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.348808 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 20:13:27.391530 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.391504 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 20:13:27.447103 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.444957 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:27.447103 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.445325 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:27.447103 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.445605 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:27.498360 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.498263 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:27.616476 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.616425 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 20:13:27.616782 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.616738 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 20:13:27.643548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.643429 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:13:27.643548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.643488 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 20:13:27.739247 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.737070 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:27.739247 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.737930 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:27.856439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.854977 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:27.856439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:27.855318 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 02 20:13:28.003924 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.003525 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 02 20:13:28.127100 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.126980 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:28.223293 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.223252 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:13:28.223567 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.223539 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 20:13:28.268412 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.268281 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:13:28.268412 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.268345 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 20:13:28.390959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.390011 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 20:13:28.395749 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.394985 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 20:13:28.396273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.395975 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 20:13:28.448926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.448014 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:28.448926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.448455 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 20:13:28.448926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.448684 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 02 20:13:28.519963 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.519923 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:28.621895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.616252 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 02 20:13:28.621895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.616314 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 02 20:13:28.633895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.623966 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:13:28.633895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.624015 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 20:13:28.747020 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.746968 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:28.747632 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.747264 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:28.857028 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.856958 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:28.857648 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.857270 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 20:13:28.981830 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:28.981796 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:13:29.005092 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.004980 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:29.019997 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.019959 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:13:29.020254 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.020231 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:29.025183 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.025155 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:13:29.071483 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.071417 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:29.092221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.092190 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:29.093634 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.093612 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:29.109619 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.109587 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:13:29.133017 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.132972 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 20:13:29.222264 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.222225 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:13:29.225995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.225942 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 20:13:29.304503 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.283716 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:13:29.304769 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.304732 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 20:13:29.386931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.359964 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:13:29.386931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.360034 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:29.395933 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.394165 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:29.536901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.535831 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:29.649422 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.649317 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:13:29.649842 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.649807 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 20:13:29.656176 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.656149 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:13:29.657181 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.657157 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:29.669484 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.669358 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 20:13:29.689283 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.689243 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:29.689961 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.689941 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:13:29.700403 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.700358 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 20:13:29.712899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.709782 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 02 20:13:29.790910 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.763737 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 20:13:29.793546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.793509 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:13:29.865405 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.863705 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:29.997151 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:29.997085 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 02 20:13:30.058935 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.056972 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 02 20:13:30.093517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.086327 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 20:13:30.246964 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.246926 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:13:30.247294 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.247235 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 02 20:13:30.318528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.318495 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:13:30.319368 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.319340 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:13:30.319574 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.319547 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 20:13:30.400112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.400060 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 20:13:30.520712 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.520635 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:13:30.524214 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.524180 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:13:30.527591 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.527568 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:13:30.620167 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.618669 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:30.621524 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.620391 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:30.628739 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.628712 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:13:30.628930 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.628904 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 20:13:30.632122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.632101 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:13:30.644569 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.644548 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:13:30.655505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.655475 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:13:30.675911 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.672758 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 20:13:30.676928 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.676903 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:13:30.741257 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.741229 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:13:30.760898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:30.760216 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:13:31.413056 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.413021 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:31.414353 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.414327 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:31.419605 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.417945 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:13:31.777286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.777207 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:31.778920 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.778061 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:31.781153 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.780408 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:13:31.787029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.787005 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx"] Nov 02 20:13:31.798168 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.798005 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.841400 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.840615 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx"] Nov 02 20:13:31.883847 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.883816 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.884380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.884351 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.884591 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.884573 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzsrj\" (UniqueName: \"kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-kube-api-access-wzsrj\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.884795 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.884775 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.884990 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.884971 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.885166 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.885149 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.885368 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.885351 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.885546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.885513 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.885813 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.885794 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.987082 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.987045 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.987285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.987116 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.987285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.987152 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.987285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.987183 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzsrj\" (UniqueName: \"kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-kube-api-access-wzsrj\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.987995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.987807 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.987995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.987860 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.987995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.987929 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.989015 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.988325 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.989015 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.988374 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.989015 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.988506 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.989015 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.988756 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.992205 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.992106 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.992324 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.992291 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.992584 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.992532 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.992681 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.992611 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:31.993643 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:31.993621 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:32.041518 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:32.041428 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:32.042122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:32.042094 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzsrj\" (UniqueName: \"kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-kube-api-access-wzsrj\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:32.108671 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:32.108641 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:32.110183 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:32.110001 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:32.112334 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:32.112312 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:13:32.119557 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:32.119535 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:32.559942 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:32.559911 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx"] Nov 02 20:13:32.568101 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:13:32.568047 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5d96d4_6d30_4ef3_9a1d_5c384355d899.slice/crio-1dfca6473eeace1cfd02b6389a51e5f1a0e2eaee8085aba3ec2c4908f927610b WatchSource:0}: Error finding container 1dfca6473eeace1cfd02b6389a51e5f1a0e2eaee8085aba3ec2c4908f927610b: Status 404 returned error can't find the container with id 1dfca6473eeace1cfd02b6389a51e5f1a0e2eaee8085aba3ec2c4908f927610b Nov 02 20:13:32.570838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:32.570803 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:32.570960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:32.570890 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:32.570960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:32.570930 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:33.224750 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.224714 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" event={"ID":"2e5d96d4-6d30-4ef3-9a1d-5c384355d899","Type":"ContainerStarted","Data":"6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474"} Nov 02 20:13:33.224952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.224766 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" event={"ID":"2e5d96d4-6d30-4ef3-9a1d-5c384355d899","Type":"ContainerStarted","Data":"1dfca6473eeace1cfd02b6389a51e5f1a0e2eaee8085aba3ec2c4908f927610b"} Nov 02 20:13:33.360898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.357079 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" podStartSLOduration=2.357057152 podStartE2EDuration="2.357057152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:33.341541855 +0000 UTC m=+1250.149326074" watchObservedRunningTime="2026-11-02 20:13:33.357057152 +0000 UTC m=+1250.164841369" Nov 02 20:13:33.385897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.383385 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw"] Nov 02 20:13:33.396160 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.396128 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.412487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.411914 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.412487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.411974 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.412487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.412005 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.412487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.412079 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.412487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.412107 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.412487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.412131 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5c7b\" (UniqueName: \"kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-kube-api-access-w5c7b\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.412487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.412226 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.412487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.412261 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.412487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.412287 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.426345 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.426305 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw"] Nov 02 20:13:33.513605 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.513521 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.513605 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.513562 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.513845 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.513688 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.513845 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.513780 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.513845 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.513821 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.514025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.513846 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5c7b\" (UniqueName: \"kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-kube-api-access-w5c7b\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.514025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.513951 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.514025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.513985 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.514025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.514001 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.514025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.514011 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.514599 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.514545 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.515020 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.514985 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.515128 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.515057 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.515385 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.515365 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.516202 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.516180 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.516391 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.516374 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.569897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.569851 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5c7b\" (UniqueName: \"kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-kube-api-access-w5c7b\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.570353 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.570100 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.655131 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.654367 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:33.724571 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:33.724541 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:33.791931 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:13:33.791896 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:13:34.120620 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:34.120534 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:34.122268 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:34.122238 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:13:34.122399 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:34.122302 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" podUID="2e5d96d4-6d30-4ef3-9a1d-5c384355d899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 20:13:34.193223 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:34.193191 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw"] Nov 02 20:13:34.198733 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:13:34.198704 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abaaa8a_f947_4d2a_b73a_424c4ff05f84.slice/crio-1c2b00e8e165816fc90632ebc3992462b1c36f8cd7c0efe83ba659ced3221076 WatchSource:0}: Error finding container 1c2b00e8e165816fc90632ebc3992462b1c36f8cd7c0efe83ba659ced3221076: Status 404 returned error can't find the container with id 1c2b00e8e165816fc90632ebc3992462b1c36f8cd7c0efe83ba659ced3221076 Nov 02 20:13:34.201298 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:34.201265 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:34.201371 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:34.201336 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:34.201371 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:34.201364 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:13:34.251439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:34.251360 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" event={"ID":"3abaaa8a-f947-4d2a-b73a-424c4ff05f84","Type":"ContainerStarted","Data":"1c2b00e8e165816fc90632ebc3992462b1c36f8cd7c0efe83ba659ced3221076"} Nov 02 20:13:35.120608 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:35.120570 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:13:35.121073 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:35.120625 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" podUID="2e5d96d4-6d30-4ef3-9a1d-5c384355d899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 20:13:35.275074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:35.275030 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" event={"ID":"3abaaa8a-f947-4d2a-b73a-424c4ff05f84","Type":"ContainerStarted","Data":"1168528efc2ccb9dfb3129b4326a2bbaf9cde2b10857cb6c7bfac11a38e6a1c1"} Nov 02 20:13:35.387016 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:35.386897 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" podStartSLOduration=2.386862102 podStartE2EDuration="2.386862102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:35.386311422 +0000 UTC m=+1252.194095643" watchObservedRunningTime="2026-11-02 20:13:35.386862102 +0000 UTC m=+1252.194646320" Nov 02 20:13:35.725679 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:35.725613 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:35.726797 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:35.726768 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:13:35.726899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:35.726828 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 20:13:36.120115 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:36.120015 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:13:36.120311 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:36.120097 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" podUID="2e5d96d4-6d30-4ef3-9a1d-5c384355d899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 20:13:36.725053 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:36.725017 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:13:36.725456 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:36.725070 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 20:13:36.884895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:36.884410 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:37.120005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:37.119920 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:13:37.120005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:37.119971 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" podUID="2e5d96d4-6d30-4ef3-9a1d-5c384355d899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 20:13:37.725719 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:37.725682 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:13:37.726229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:37.725730 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 20:13:38.120911 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:38.120803 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:13:38.120911 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:38.120855 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" podUID="2e5d96d4-6d30-4ef3-9a1d-5c384355d899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 20:13:38.503122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:38.503083 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:38.725395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:38.725358 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:13:38.725567 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:38.725417 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 20:13:39.120234 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:39.120184 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:13:39.120737 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:39.120261 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" podUID="2e5d96d4-6d30-4ef3-9a1d-5c384355d899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 20:13:39.725242 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:39.725197 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:13:39.725462 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:39.725258 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 20:13:40.356897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:40.344215 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:40.356897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:40.345720 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:40.356897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:40.353131 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:13:40.731630 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:40.731589 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:13:40.731926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:40.731869 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 20:13:42.412070 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:42.410003 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:42.505035 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:42.504973 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:42.512184 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:42.512140 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:42.646030 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:42.645977 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:42.720449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:42.720344 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:42.721657 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:42.721615 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 20:13:42.774566 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:42.774143 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:13:42.774566 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:42.774208 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:43.001698 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.001603 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:43.003463 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.003442 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:43.003810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.003782 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:43.011129 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.011109 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:13:43.051897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.049962 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:43.188823 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.188777 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:43.291404 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.291282 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:43.291791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.291763 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 20:13:43.293914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.293869 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 02 20:13:43.314898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.304001 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:43.328408 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.328365 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:43.507718 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.506946 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:43.515038 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.514951 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:43.526112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.525581 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:43.541947 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.538911 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:13:43.541947 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.539309 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:13:43.596839 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.596809 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:13:43.605613 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.605587 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:13:43.620864 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:43.617482 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:47.822539 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:47.822508 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:48.111060 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:48.110944 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:52.665288 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:52.665233 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:13:57.254906 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:13:57.254808 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:01.882215 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:01.882164 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:02.624586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.624556 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6"] Nov 02 20:14:02.628497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.628474 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.691590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.691553 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.691780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.691596 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh72m\" (UniqueName: \"kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-kube-api-access-bh72m\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.691780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.691670 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.691893 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.691765 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88985b8c-444e-4b57-b179-2e114dfa7d48-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.691893 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.691801 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.691893 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.691817 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.692000 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.691932 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.692000 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.691960 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.692108 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.692070 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.720209 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.720169 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6"] Nov 02 20:14:02.793427 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.793388 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.793606 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.793537 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.793606 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.793572 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh72m\" (UniqueName: \"kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-kube-api-access-bh72m\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.793730 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.793654 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.793730 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.793716 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88985b8c-444e-4b57-b179-2e114dfa7d48-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.793906 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.793859 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.794104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.794078 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.794211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.794155 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.794211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.794204 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.794374 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.794346 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.794604 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.794577 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88985b8c-444e-4b57-b179-2e114dfa7d48-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.794677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.794619 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.794677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.794645 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.794795 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.794689 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.796372 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.796352 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.796847 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.796831 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.847714 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.847681 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.848091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.848068 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh72m\" (UniqueName: \"kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-kube-api-access-bh72m\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:02.942771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:02.942731 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:03.371994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:03.371911 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6"] Nov 02 20:14:03.379536 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:14:03.379509 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88985b8c_444e_4b57_b179_2e114dfa7d48.slice/crio-1a309b71bfd4be8488ef05db3105f9850d726692bfb6226719578618374db563 WatchSource:0}: Error finding container 1a309b71bfd4be8488ef05db3105f9850d726692bfb6226719578618374db563: Status 404 returned error can't find the container with id 1a309b71bfd4be8488ef05db3105f9850d726692bfb6226719578618374db563 Nov 02 20:14:03.382027 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:03.381993 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:03.382096 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:03.382067 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:03.382132 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:03.382098 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:03.527284 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:03.527245 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" event={"ID":"88985b8c-444e-4b57-b179-2e114dfa7d48","Type":"ContainerStarted","Data":"f5df2de9ae8ae1df2d32c983e0670cd2e5fdd26b3982fb77d77f2bb3914131a0"} Nov 02 20:14:03.527487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:03.527307 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" event={"ID":"88985b8c-444e-4b57-b179-2e114dfa7d48","Type":"ContainerStarted","Data":"1a309b71bfd4be8488ef05db3105f9850d726692bfb6226719578618374db563"} Nov 02 20:14:03.638640 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:03.638536 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" podStartSLOduration=1.638522265 podStartE2EDuration="1.638522265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:03.637087612 +0000 UTC m=+1280.444871840" watchObservedRunningTime="2026-11-02 20:14:03.638522265 +0000 UTC m=+1280.446306480" Nov 02 20:14:03.782792 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:14:03.782760 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:14:03.943043 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:03.943012 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:04.951901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:04.951008 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 20:14:04.951901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:04.951077 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 20:14:05.945898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:05.945286 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 20:14:05.945898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:05.945349 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 20:14:06.414201 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:06.414168 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:06.950900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:06.950288 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 20:14:06.950900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:06.950369 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 20:14:07.646855 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:07.646769 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:07.955309 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:07.953532 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 20:14:07.955309 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:07.953618 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 20:14:08.943261 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:08.943222 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 20:14:08.943905 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:08.943851 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 20:14:09.944283 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:09.944235 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 20:14:09.945164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:09.945124 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 20:14:10.599026 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:10.598969 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:10.646213 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:10.646164 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:10.958763 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:10.958278 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 20:14:10.958763 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:10.958356 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 20:14:11.046766 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:11.046386 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 20:14:11.124792 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:11.124745 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 20:14:11.125429 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:11.125396 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:11.138044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:11.137997 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:11.183913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:11.182007 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:11.882302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:11.882256 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:11.945089 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:11.944998 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:12.988141 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:12.988092 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:14:12.988772 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:12.988739 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:13.509037 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:13.509002 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:13.509530 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:13.509499 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:13.510490 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:13.510469 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:14:16.673951 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:16.673865 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:21.164328 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:21.164292 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:25.661321 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:25.661284 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:30.268704 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:30.268653 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:31.883462 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:31.883426 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5"] Nov 02 20:14:31.887291 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:31.887273 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:31.936938 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:31.936908 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9fcedd-openshift-default-dockercfg-rmhls\"" Nov 02 20:14:31.988273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:31.988240 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5"] Nov 02 20:14:32.058070 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.058033 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba153ce-9124-483e-b2d4-ae582cca5f6b-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.058264 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.058091 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz8qd\" (UniqueName: \"kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-kube-api-access-mz8qd\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.058264 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.058192 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.058264 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.058228 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.058264 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.058252 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.058498 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.058472 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.058594 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.058576 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.058632 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.058622 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.058675 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.058659 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.159495 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.159463 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.159686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.159504 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.159686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.159574 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba153ce-9124-483e-b2d4-ae582cca5f6b-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.159686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.159612 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz8qd\" (UniqueName: \"kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-kube-api-access-mz8qd\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.159686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.159658 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.159929 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.159692 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.159929 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.159712 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.159929 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.159913 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.160093 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.160018 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.160158 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.160096 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.160359 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.160335 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.160446 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.160371 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.160446 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.160399 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.160527 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.160443 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba153ce-9124-483e-b2d4-ae582cca5f6b-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.162053 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.162031 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.162255 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.162233 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.214708 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.214671 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz8qd\" (UniqueName: \"kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-kube-api-access-mz8qd\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.214930 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.214911 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:32.504272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:32.504179 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:33.159933 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.144097 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5"] Nov 02 20:14:33.211342 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.211307 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd"] Nov 02 20:14:33.232914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.230762 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:33.232914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.230888 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:33.232914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.230943 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:33.249899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.239623 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.290203 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.284636 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd"] Nov 02 20:14:33.410972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.389099 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.410972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.408166 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.410972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.408233 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st982\" (UniqueName: \"kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-kube-api-access-st982\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.410972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.408269 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.410972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.408312 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.410972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.408357 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.410972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.408471 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.410972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.408512 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.410972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.408572 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.597387 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.597348 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.764313 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.600127 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.773856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.768098 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st982\" (UniqueName: \"kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-kube-api-access-st982\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.793012 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.792975 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.793285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.793261 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.793440 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.793421 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.793666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.793643 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.793833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.793813 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.794024 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.794005 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.805906 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.794237 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.807898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.806422 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.808062 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.807106 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.826621 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.826589 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.838707 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.838672 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.922550 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.922512 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.932942 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.932190 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.932942 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.932460 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:33.966900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:33.951670 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st982\" (UniqueName: \"kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-kube-api-access-st982\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:34.232047 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.211378 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:34.232047 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.211796 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:34.232047 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.212173 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 02 20:14:34.232047 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.212486 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 20:14:34.232047 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.214615 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:34.232047 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.214934 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 20:14:34.283503 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.258528 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" event={"ID":"cba153ce-9124-483e-b2d4-ae582cca5f6b","Type":"ContainerStarted","Data":"7cb8ad37634a4c7fe156d86c6cf24fd42b6325cc679be4ed31117473ecd5b089"} Nov 02 20:14:34.310750 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:14:34.303121 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:14:34.366238 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.337830 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:34.717979 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.717544 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:34.737934 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.718749 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 20:14:34.748266 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.745603 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Nov 02 20:14:34.793901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.753456 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 20:14:34.793901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:34.754800 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 02 20:14:35.009438 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.009335 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.047367 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.033226 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.066052 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.064930 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.219901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.213959 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.219901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.217474 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 20:14:35.219901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.217808 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 20:14:35.219901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.217951 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 02 20:14:35.219901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.218362 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:14:35.219901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.218423 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:14:35.219901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.218460 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:14:35.219901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.218495 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:14:35.220916 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.220864 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 02 20:14:35.221045 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.221011 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:14:35.297365 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.297252 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" event={"ID":"cba153ce-9124-483e-b2d4-ae582cca5f6b","Type":"ContainerStarted","Data":"7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786"} Nov 02 20:14:35.347023 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.346973 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.392906 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.379304 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:14:35.557898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.557797 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:35.558193 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.558168 5561 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:14:35.558326 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.558305 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 20:14:35.740784 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.739996 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 20:14:35.770260 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.769163 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.812957 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.771411 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.812957 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.778052 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 20:14:35.812957 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.778316 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 20:14:35.812957 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.794515 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:14:35.812957 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.794582 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:14:35.812957 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.794620 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:14:35.812957 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.794655 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:14:35.812957 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.794694 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:14:35.891677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:35.891635 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Nov 02 20:14:36.039996 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.039950 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.170898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.168450 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:14:36.170898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.066044 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.170898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.169558 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:14:36.215901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.192584 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" podStartSLOduration=5.192563617 podStartE2EDuration="5.192563617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:36.04046183 +0000 UTC m=+1312.848246090" watchObservedRunningTime="2026-11-02 20:14:36.192563617 +0000 UTC m=+1313.000347834" Nov 02 20:14:36.218232 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.218199 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.218448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.217685 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.218621 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.215375 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.243661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.242469 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd"] Nov 02 20:14:36.262508 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.240965 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.265636 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.264130 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 20:14:36.274899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.266392 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:14:36.279833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.279176 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:36.279833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.279280 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:36.279833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.279331 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:36.301908 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.298574 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:14:36.335902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.332596 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:14:36.363923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.351597 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.442956 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.439183 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" event={"ID":"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c","Type":"ContainerStarted","Data":"3a42ae9f807a83dad63e14e9ecaba435f8da737096f906d81a317c3bdbcfcbcf"} Nov 02 20:14:36.463309 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.447348 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.694725 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.694638 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:14:36.712784 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.712737 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Nov 02 20:14:36.736423 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.736386 5561 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:14:36.736733 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.736683 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 20:14:36.745719 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.745687 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 20:14:36.775359 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.775308 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.783139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.783091 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 20:14:36.827225 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.827188 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:14:36.836438 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.836416 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:14:36.866348 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.858549 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.926428 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.913691 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:14:36.926428 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.926085 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:14:36.926428 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.926143 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:14:36.926744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:36.926689 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:14:37.015527 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:37.015222 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:14:37.022496 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:37.022470 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:14:37.022736 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:37.022719 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:14:37.469110 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:37.469063 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" event={"ID":"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c","Type":"ContainerStarted","Data":"10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b"} Nov 02 20:14:37.505732 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:37.505694 5561 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:14:37.505914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:37.505756 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 20:14:38.213304 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:38.213270 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:38.214425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:38.214390 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:14:38.214558 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:38.214447 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" podUID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 20:14:38.505092 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:38.505021 5561 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:14:38.505585 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:38.505561 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 20:14:39.200357 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:39.200303 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" podStartSLOduration=6.200286408 podStartE2EDuration="6.200286408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:39.197582211 +0000 UTC m=+1316.005366427" watchObservedRunningTime="2026-11-02 20:14:39.200286408 +0000 UTC m=+1316.008070625" Nov 02 20:14:39.212252 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:39.212199 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:14:39.212416 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:39.212272 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" podUID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 20:14:39.505727 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:39.505642 5561 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:14:39.505727 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:39.505689 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 20:14:39.960832 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:39.960794 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:40.213078 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:40.212991 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:14:40.213078 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:40.213046 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" podUID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 20:14:40.505406 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:40.505317 5561 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:14:40.505549 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:40.505404 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 20:14:41.212943 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:41.212907 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:14:41.213422 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:41.212977 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" podUID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 20:14:41.505777 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:41.505700 5561 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:14:41.505777 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:41.505766 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 20:14:41.911894 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:41.911829 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:42.212607 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:42.212517 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:14:42.212783 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:42.212596 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" podUID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 20:14:42.504702 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:42.504615 5561 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:14:42.504702 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:42.504682 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 20:14:43.212568 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:43.212529 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:14:43.212751 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:43.212597 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" podUID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 20:14:43.619893 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:43.619392 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:43.621034 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:43.620998 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:43.622469 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:43.622034 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:14:44.212664 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:44.212621 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:14:44.212844 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:44.212706 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" podUID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 20:14:45.216901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:45.216856 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:45.217341 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:45.217288 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:45.218089 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:45.218065 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:14:46.545256 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:46.545220 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:48.162401 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.162368 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:48.256325 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.256285 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz"] Nov 02 20:14:48.261136 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.261116 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.353905 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.353841 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz"] Nov 02 20:14:48.390443 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.390401 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.390619 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.390577 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6h2z\" (UniqueName: \"kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-kube-api-access-j6h2z\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.390678 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.390624 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.390810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.390785 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.390908 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.390890 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.391079 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.391059 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.391131 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.391100 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.391188 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.391173 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.391260 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.391238 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492191 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492110 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492364 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492202 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492364 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492223 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492364 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492246 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492364 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492306 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492364 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492338 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492632 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492438 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6h2z\" (UniqueName: \"kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-kube-api-access-j6h2z\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492632 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492481 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492632 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492548 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492632 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492612 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492675 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492687 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.492959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.492915 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.493265 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.493242 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.494692 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.494667 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.494841 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.494825 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.495980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.495953 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:48.551405 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.551370 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.551726 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.551702 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6h2z\" (UniqueName: \"kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-kube-api-access-j6h2z\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:48.574212 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:48.574186 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:49.005478 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:49.005443 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz"] Nov 02 20:14:49.012291 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:14:49.012254 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae3c417_3c1b_4e3f_8ed8_ce611c9c6c0b.slice/crio-fcfb888161a239b7e3a34d8d1b89e4811443fa7b91fadd44317b45a936e92160 WatchSource:0}: Error finding container fcfb888161a239b7e3a34d8d1b89e4811443fa7b91fadd44317b45a936e92160: Status 404 returned error can't find the container with id fcfb888161a239b7e3a34d8d1b89e4811443fa7b91fadd44317b45a936e92160 Nov 02 20:14:49.014728 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:49.014691 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:49.014827 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:49.014774 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:49.014919 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:49.014848 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:14:49.572727 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:49.572692 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" event={"ID":"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b","Type":"ContainerStarted","Data":"f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6"} Nov 02 20:14:49.573157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:49.572745 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" event={"ID":"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b","Type":"ContainerStarted","Data":"fcfb888161a239b7e3a34d8d1b89e4811443fa7b91fadd44317b45a936e92160"} Nov 02 20:14:49.575066 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:49.575044 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:49.683268 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:49.683219 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" podStartSLOduration=1.6832056949999998 podStartE2EDuration="1.683205695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:49.682635885 +0000 UTC m=+1326.490420119" watchObservedRunningTime="2026-11-02 20:14:49.683205695 +0000 UTC m=+1326.490989914" Nov 02 20:14:50.575901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:50.575845 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 20:14:50.576306 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:50.575943 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" podUID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 20:14:51.270994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:51.270962 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:51.575237 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:51.575148 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 20:14:51.575237 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:51.575203 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" podUID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 20:14:52.574706 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:52.574670 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 20:14:52.575139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:52.574727 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" podUID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 20:14:53.575436 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:53.575322 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 20:14:53.575436 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:53.575376 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" podUID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 20:14:53.659297 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:53.659252 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:53.870965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:53.870865 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:54.400293 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:54.400252 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:54.574790 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:54.574755 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 20:14:54.574995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:54.574808 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" podUID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 20:14:55.575605 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:55.575565 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 20:14:55.576057 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:55.575620 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" podUID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 20:14:55.961746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:55.961708 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:56.579184 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:56.579152 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:56.579633 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:56.579562 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:14:56.580395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:14:56.580377 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:15:00.017321 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:00.017285 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:00.203757 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:00.203726 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:00.391390 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:00.391303 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:00.573963 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:00.573933 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:01.316722 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:01.316686 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:01.507604 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:01.507558 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:01.698531 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:01.698495 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:02.514224 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:02.514193 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:02.693923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:02.693885 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:03.787996 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:15:03.787950 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:15:05.346340 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:05.346304 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:05.661433 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:05.661316 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:05.849446 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:05.849399 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:06.056488 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:06.056413 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:06.239563 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:06.239510 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:06.418314 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:06.418279 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:06.831383 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:06.831301 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:07.018908 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:07.018858 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:07.190395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:07.190331 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:09.837098 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:09.837053 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:10.117962 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.110133 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:10.201042 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.200994 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.227994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.227948 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.229365 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.228263 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 20:15:10.229690 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.228936 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 20:15:10.309290 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.309241 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.344901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.342979 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.884903 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.837012 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.884903 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.880268 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 02 20:15:10.884903 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.880398 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded" Nov 02 20:15:10.884903 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.880563 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.951763 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.951713 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.185960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.164186 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 20:15:11.185960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:10.960963 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.311902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.201504 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.311902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.208314 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:15:11.311902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.233698 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.311902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.234124 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 02 20:15:11.311902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.234434 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:15:11.311902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.234491 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:15:11.311902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.254037 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.311902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.276211 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:15:11.352207 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.314784 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.352207 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.352052 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.634318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.634224 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:15:11.658060 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.658020 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:15:11.807024 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.806975 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.873781 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.873739 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 20:15:11.874586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.874564 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:15:11.889690 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.889591 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.890621 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.890588 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 02 20:15:11.891092 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.891073 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:15:11.891263 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.891249 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:15:11.911614 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.911519 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded" Nov 02 20:15:11.921427 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:11.921391 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.062924 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.059989 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:12.220083 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.201430 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.220083 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.201793 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 20:15:12.220083 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.202129 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:15:12.220083 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.202186 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:15:12.221133 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.221093 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.221657 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.221632 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 02 20:15:12.222145 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.222126 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:15:12.275891 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.248587 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.305333 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.248622 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 02 20:15:12.319032 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.260184 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.326901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.314948 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 20:15:12.471956 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.460506 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 20:15:12.528932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.528892 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.632085 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.632048 5561 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:15:12.632364 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.632332 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" podUID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.937525 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.937471 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.938589 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.938541 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.969038 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.938754 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.969611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.939237 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.970185 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:12.970164 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:15:13.203292 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.203187 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 20:15:13.234902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.204756 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.234902 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.230787 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 02 20:15:13.258900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.250011 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.287341 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.274997 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 02 20:15:13.318126 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.318078 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.332684 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.332639 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.333696 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.333662 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.334177 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.334148 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.745891 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.743443 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Nov 02 20:15:13.807161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.807125 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:13.813999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.813964 5561 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.817544 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.817521 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:15:13.817846 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.817831 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:15:13.818051 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.818018 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:15:13.818205 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.818192 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:15:13.848914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.848363 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:15:13.876018 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.875218 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:15:13.902660 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.902632 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" Nov 02 20:15:13.940030 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.939986 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 02 20:15:13.990191 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:13.988471 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:15:14.007428 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:14.007344 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:15:14.205084 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:14.205043 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 02 20:15:14.370332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:14.342763 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:14.374921 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:14.342799 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 20:15:14.568054 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:14.550590 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:15:14.661961 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:14.652022 5561 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:15:14.661961 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:14.652084 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:14.724125 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:14.723194 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:15:14.988970 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:14.977393 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 02 20:15:15.442026 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:15.441978 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 20:15:15.678906 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:15.678859 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:15:15.679193 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:15.679174 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:15:17.664387 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:17.664347 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:17.886674 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:17.886630 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:18.059693 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:18.059602 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:18.888369 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:18.888329 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:20.144937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:20.144893 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:20.330817 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:20.330785 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:21.455193 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:21.455146 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:21.648769 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:21.648735 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:23.664471 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:23.664435 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:23.930033 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:23.929913 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:24.155487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:24.155454 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:26.548646 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.548616 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr"] Nov 02 20:15:26.553754 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.553736 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.610137 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.610107 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 20:15:26.653076 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.653043 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0284d6e4-70b6-467c-a101-1872f264a098-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.653212 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.653141 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.653248 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.653234 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.653347 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.653255 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.653347 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.653277 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.653466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.653439 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 20:15:26.653466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.653447 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-42zng\"" Nov 02 20:15:26.653466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.653442 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 20:15:26.653560 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.653462 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m628t\" (UniqueName: \"kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-kube-api-access-m628t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.653560 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.653498 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.664532 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.664503 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr"] Nov 02 20:15:26.754318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.754282 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m628t\" (UniqueName: \"kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-kube-api-access-m628t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.754498 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.754323 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.754498 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.754376 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0284d6e4-70b6-467c-a101-1872f264a098-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.754498 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.754409 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.754498 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.754493 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.754720 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.754619 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.754720 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.754651 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.755386 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.755357 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.756801 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.756772 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0284d6e4-70b6-467c-a101-1872f264a098-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.756983 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.756963 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.757051 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.756981 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.757051 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.756996 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.809247 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.809180 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.809247 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.809212 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m628t\" (UniqueName: \"kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-kube-api-access-m628t\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4lhbr\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:26.867432 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:26.867399 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:27.167121 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:27.167055 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:27.294658 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:27.294625 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr"] Nov 02 20:15:27.300983 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:15:27.300951 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0284d6e4_70b6_467c_a101_1872f264a098.slice/crio-86896826a2798c9d5600e1bc48facb58dcffbc27a0a53ed503dfbd20ef08a945 WatchSource:0}: Error finding container 86896826a2798c9d5600e1bc48facb58dcffbc27a0a53ed503dfbd20ef08a945: Status 404 returned error can't find the container with id 86896826a2798c9d5600e1bc48facb58dcffbc27a0a53ed503dfbd20ef08a945 Nov 02 20:15:27.303505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:27.303472 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:15:27.303595 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:27.303531 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:15:27.357009 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:27.356980 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:27.491420 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:27.491317 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" event={"ID":"0284d6e4-70b6-467c-a101-1872f264a098","Type":"ContainerStarted","Data":"e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837"} Nov 02 20:15:27.491420 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:27.491372 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" event={"ID":"0284d6e4-70b6-467c-a101-1872f264a098","Type":"ContainerStarted","Data":"86896826a2798c9d5600e1bc48facb58dcffbc27a0a53ed503dfbd20ef08a945"} Nov 02 20:15:27.491659 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:27.491459 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:28.516644 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:28.516607 5561 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-4lhbr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:15:28.517205 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:28.516666 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" podUID="0284d6e4-70b6-467c-a101-1872f264a098" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:15:28.659203 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:28.659159 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:30.274261 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:30.274223 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:30.885313 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:30.885274 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:31.164951 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:31.164858 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:31.502868 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:31.502789 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:15:31.708538 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:31.708323 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" podStartSLOduration=5.708302563 podStartE2EDuration="5.708302563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:27.607167153 +0000 UTC m=+1364.414951381" watchObservedRunningTime="2026-11-02 20:15:31.708302563 +0000 UTC m=+1368.516086780" Nov 02 20:15:33.167081 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:33.167042 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:33.609021 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:33.608930 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:33.780334 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:15:33.780299 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:15:35.061796 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:35.061761 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:35.255530 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:35.255494 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:35.427652 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:35.427621 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:38.165846 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:38.165766 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:38.349888 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:38.349847 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:38.612567 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:38.612485 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:39.743630 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:39.743591 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:39.922134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:39.922103 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:43.158645 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:43.158604 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:46.364092 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:46.364025 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:47.803895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:47.803647 5561 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:15:47.803895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:47.803735 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:48.327012 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:48.326961 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:48.387003 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:48.386957 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:48.649030 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:48.648939 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:48.808041 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:48.807995 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:48.937751 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:48.937308 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x" podUID="e8b4ee4a-ebb3-4872-bb79-520fec0376ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:49.127436 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:49.127398 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:49.317708 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:49.317635 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:50.583502 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:50.583460 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:54.112233 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:54.112191 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:54.574169 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:54.574128 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:54.748686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:54.748643 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:58.561120 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:15:58.561081 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:03.159315 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:03.159262 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:03.780733 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:16:03.780695 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:16:07.695252 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:07.695184 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:12.103800 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:12.103762 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:12.277284 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:12.277245 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:12.829914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:12.829851 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:16.853086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:16.853051 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:19.760470 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.760429 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s"] Nov 02 20:16:19.764657 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.764625 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:19.815061 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.815026 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fdd8j\"" Nov 02 20:16:19.867342 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.867297 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s"] Nov 02 20:16:19.937144 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.937104 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:19.937316 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.937163 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:19.937316 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.937248 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa33f7d5-310c-4581-84c4-317226e57817-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:19.937316 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.937280 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rznlx\" (UniqueName: \"kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-kube-api-access-rznlx\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:19.937481 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.937453 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:19.937542 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.937511 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:19.937595 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.937551 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:19.937724 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.937702 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:19.937762 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:19.937748 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa33f7d5-310c-4581-84c4-317226e57817-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.039273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.039197 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.039273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.039236 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa33f7d5-310c-4581-84c4-317226e57817-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.039508 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.039293 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.039508 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.039319 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.039508 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.039351 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa33f7d5-310c-4581-84c4-317226e57817-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.039508 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.039383 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rznlx\" (UniqueName: \"kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-kube-api-access-rznlx\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.039508 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.039450 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.039508 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.039500 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.039869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.039541 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.039869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.039775 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.040029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.039901 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.040029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.039999 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.040029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.040018 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.040186 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.040093 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa33f7d5-310c-4581-84c4-317226e57817-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.041516 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.041494 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.041736 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.041719 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa33f7d5-310c-4581-84c4-317226e57817-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.093652 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.093617 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.093797 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.093757 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rznlx\" (UniqueName: \"kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-kube-api-access-rznlx\") pod \"maas-default-gateway-openshift-default-696f47c6c-pbl5s\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.378718 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.378613 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:20.818685 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.818643 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s"] Nov 02 20:16:20.823471 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:16:20.823443 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa33f7d5_310c_4581_84c4_317226e57817.slice/crio-107f1080b97af2cb5e9970473279820f57243e8f80f114b15c2ce5e04c516192 WatchSource:0}: Error finding container 107f1080b97af2cb5e9970473279820f57243e8f80f114b15c2ce5e04c516192: Status 404 returned error can't find the container with id 107f1080b97af2cb5e9970473279820f57243e8f80f114b15c2ce5e04c516192 Nov 02 20:16:20.826073 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.826036 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:16:20.826141 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.826108 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:16:20.826141 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.826136 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:16:20.890949 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:20.890912 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" event={"ID":"fa33f7d5-310c-4581-84c4-317226e57817","Type":"ContainerStarted","Data":"107f1080b97af2cb5e9970473279820f57243e8f80f114b15c2ce5e04c516192"} Nov 02 20:16:21.457077 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:21.457044 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:21.898466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:21.898381 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" event={"ID":"fa33f7d5-310c-4581-84c4-317226e57817","Type":"ContainerStarted","Data":"20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7"} Nov 02 20:16:22.105271 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:22.105220 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" podStartSLOduration=3.105203561 podStartE2EDuration="3.105203561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:22.058054655 +0000 UTC m=+1418.865838928" watchObservedRunningTime="2026-11-02 20:16:22.105203561 +0000 UTC m=+1418.912987778" Nov 02 20:16:22.378971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:22.378934 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:22.380263 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:22.380233 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-pbl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:16:22.380382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:22.380286 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" podUID="fa33f7d5-310c-4581-84c4-317226e57817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 20:16:23.379958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:23.379919 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-pbl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:16:23.380370 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:23.379980 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" podUID="fa33f7d5-310c-4581-84c4-317226e57817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 20:16:24.379429 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:24.379396 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-pbl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:16:24.379638 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:24.379450 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" podUID="fa33f7d5-310c-4581-84c4-317226e57817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 20:16:25.379268 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:25.379231 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-pbl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:16:25.379723 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:25.379305 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" podUID="fa33f7d5-310c-4581-84c4-317226e57817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 20:16:25.960861 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:25.960665 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:26.378976 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:26.378863 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-pbl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:16:26.378976 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:26.378958 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" podUID="fa33f7d5-310c-4581-84c4-317226e57817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 20:16:27.379272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:27.379234 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-pbl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:16:27.379687 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:27.379293 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" podUID="fa33f7d5-310c-4581-84c4-317226e57817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 20:16:28.379204 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:28.379167 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-pbl5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:16:28.379409 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:28.379230 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" podUID="fa33f7d5-310c-4581-84c4-317226e57817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 20:16:29.613661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:29.613628 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:29.614220 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:29.614087 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:29.615001 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:29.614978 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:16:30.455285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:30.455241 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:32.697595 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:32.697561 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:33.780283 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:16:33.780244 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:16:35.128552 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:35.128498 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:39.621282 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:39.621241 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:44.142318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:44.142280 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:48.673132 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:48.673090 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:49.051449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:49.051348 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:49.322697 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:49.322601 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:53.320347 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:53.320310 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:58.333740 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:16:58.333640 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:02.974765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:02.974722 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:03.772728 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:03.772696 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:03.780673 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:17:03.780639 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:17:03.955861 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:03.955816 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:06.044867 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:06.044821 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:06.249468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:06.249434 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:07.550034 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:07.549998 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:09.000611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:09.000573 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:09.179282 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:09.179243 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:12.087805 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:12.087770 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:13.253186 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:13.253151 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:13.425219 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:13.425180 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:14.764591 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:14.764553 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:16.903647 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:16.903610 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:21.258204 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:21.258166 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:21.699209 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:21.699171 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:26.455993 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:26.455943 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:27.133094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:27.133052 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:28.298410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:28.298378 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:28.472156 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:28.472120 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:28.648233 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:28.648153 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:30.986112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:30.986068 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:32.634112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:32.634047 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:33.507838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:33.507806 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:33.692237 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:33.692202 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:33.783929 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:17:33.783803 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:17:34.144063 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:34.143966 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:35.842635 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:35.842590 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:39.999950 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:39.999859 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:40.171130 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:40.171097 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:41.143626 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:41.143583 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:41.500052 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:41.500018 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:41.746814 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:41.746770 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:43.211935 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:43.211900 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:43.434314 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:43.434246 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:43.606184 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:43.606087 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:43.798728 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:43.798651 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:47.108705 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:47.108658 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:47.299671 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:47.299629 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:47.995759 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:47.995719 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:49.400931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:49.400869 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:50.320897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.320849 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:50.466422 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.466390 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v"] Nov 02 20:17:50.470752 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.470724 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.506652 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.506619 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:50.529913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.529868 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v"] Nov 02 20:17:50.540549 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.540514 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ld52\" (UniqueName: \"kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-kube-api-access-5ld52\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.540722 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.540623 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.540722 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.540671 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.540814 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.540748 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.540918 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.540894 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.540991 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.540921 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a80651-6634-491d-89cb-59939a3c7ef8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.541115 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.541094 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.541182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.541146 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.541228 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.541208 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.643573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.643483 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.644295 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.644246 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a80651-6634-491d-89cb-59939a3c7ef8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.645411 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.645390 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.646001 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.645982 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.647691 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.647220 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.647946 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.646377 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.647946 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.645904 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.647946 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.645257 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a80651-6634-491d-89cb-59939a3c7ef8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.647946 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.647670 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.648401 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.648381 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ld52\" (UniqueName: \"kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-kube-api-access-5ld52\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.648943 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.648923 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.649410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.649393 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.650117 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.650100 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.650604 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.649325 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.674921 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.674847 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.675538 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.675510 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.703036 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.703006 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ld52\" (UniqueName: \"kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-kube-api-access-5ld52\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.704340 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.704313 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-8zc4v\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:50.786683 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:50.786643 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:51.227549 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:51.227518 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v"] Nov 02 20:17:51.234583 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:17:51.234552 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a80651_6634_491d_89cb_59939a3c7ef8.slice/crio-75a7ab75327e86ae6153723cba7f5581339d4b3b2413c76b8a8a462cb116e74b WatchSource:0}: Error finding container 75a7ab75327e86ae6153723cba7f5581339d4b3b2413c76b8a8a462cb116e74b: Status 404 returned error can't find the container with id 75a7ab75327e86ae6153723cba7f5581339d4b3b2413c76b8a8a462cb116e74b Nov 02 20:17:51.237151 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:51.237119 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:51.237232 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:51.237185 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:51.237232 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:51.237214 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:17:51.541433 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:51.541335 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" event={"ID":"c8a80651-6634-491d-89cb-59939a3c7ef8","Type":"ContainerStarted","Data":"d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1"} Nov 02 20:17:51.541433 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:51.541408 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" event={"ID":"c8a80651-6634-491d-89cb-59939a3c7ef8","Type":"ContainerStarted","Data":"75a7ab75327e86ae6153723cba7f5581339d4b3b2413c76b8a8a462cb116e74b"} Nov 02 20:17:51.652545 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:51.652496 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" podStartSLOduration=1.652482097 podStartE2EDuration="1.652482097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:51.651081775 +0000 UTC m=+1508.458866017" watchObservedRunningTime="2026-11-02 20:17:51.652482097 +0000 UTC m=+1508.460266312" Nov 02 20:17:51.787497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:51.787460 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:17:52.788457 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:52.788422 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8zc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:17:52.788838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:52.788489 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 20:17:52.946488 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:52.946447 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:53.787922 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:53.787853 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8zc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:17:53.788105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:53.787987 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 20:17:53.918581 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:53.918544 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:54.561102 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:54.561071 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:54.788053 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:54.788012 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8zc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:17:54.788238 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:54.788090 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 20:17:55.155002 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:55.154954 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:55.787627 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:55.787592 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8zc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:17:55.787914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:55.787857 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 20:17:56.787736 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:56.787697 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8zc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:17:56.788243 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:56.787754 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 20:17:57.159158 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:57.159103 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:57.787427 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:57.787394 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8zc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:17:57.787672 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:57.787451 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 20:17:58.083926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:58.083818 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:58.274344 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:58.274312 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:58.526641 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:58.526609 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:58.787639 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:58.787545 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8zc4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:17:58.787639 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:58.787598 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 02 20:17:58.892278 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:58.892204 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:59.147172 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:59.147090 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:59.333818 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:59.333781 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:59.521597 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:17:59.521560 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:00.070854 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:00.070817 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:18:00.071358 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:00.071330 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:18:00.072358 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:00.072341 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:18:00.791573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:00.791535 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:01.606217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:01.606181 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:01.774866 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:01.774833 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:01.946467 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:01.946436 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:03.780659 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:18:03.780621 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:18:04.268788 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:04.268742 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:05.548810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.548769 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb"] Nov 02 20:18:05.553241 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.553212 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.597396 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.597361 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb"] Nov 02 20:18:05.616372 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.616333 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.616580 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.616393 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.616580 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.616427 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7hrj\" (UniqueName: \"kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-kube-api-access-l7hrj\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.616709 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.616581 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.616709 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.616696 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.616822 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.616786 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.616910 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.616855 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.616984 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.616973 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.617039 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.617004 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.718558 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.718519 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7hrj\" (UniqueName: \"kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-kube-api-access-l7hrj\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.718744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.718588 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.718744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.718635 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.718744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.718684 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.718915 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.718750 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.718915 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.718820 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.718915 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.718857 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.718915 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.718908 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.719125 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.719105 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.719125 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.719112 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.719246 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.719180 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.719443 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.719423 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.719507 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.719486 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.719563 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.719485 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.721294 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.721268 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.721393 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.721364 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.772471 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.772442 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.772680 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.772662 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7hrj\" (UniqueName: \"kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-kube-api-access-l7hrj\") pod \"maas-default-gateway-openshift-default-696f47c6c-q5tqb\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:05.867841 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:05.867750 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:06.045848 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:06.045813 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:06.230392 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:06.230356 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:06.302189 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:06.302154 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb"] Nov 02 20:18:06.307798 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:18:06.307768 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ad491f_a6ab_40ee_8e44_e24ec9a0da3b.slice/crio-ce05e84f505c0bf0cf5cae4bdb0d7f61b1169f202b5cf6f3bab9016108e54f24 WatchSource:0}: Error finding container ce05e84f505c0bf0cf5cae4bdb0d7f61b1169f202b5cf6f3bab9016108e54f24: Status 404 returned error can't find the container with id ce05e84f505c0bf0cf5cae4bdb0d7f61b1169f202b5cf6f3bab9016108e54f24 Nov 02 20:18:06.310649 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:06.310615 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:06.310723 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:06.310687 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:06.310723 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:06.310715 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:18:06.663501 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:06.663462 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" event={"ID":"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b","Type":"ContainerStarted","Data":"ab333609cbdce4df06882eb4a69986f06a9c6b2b48b016aae0218aa554f7cac1"} Nov 02 20:18:06.664010 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:06.663534 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" event={"ID":"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b","Type":"ContainerStarted","Data":"ce05e84f505c0bf0cf5cae4bdb0d7f61b1169f202b5cf6f3bab9016108e54f24"} Nov 02 20:18:06.778325 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:06.778259 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" podStartSLOduration=1.7782389969999999 podStartE2EDuration="1.778238997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:06.77551629 +0000 UTC m=+1523.583300508" watchObservedRunningTime="2026-11-02 20:18:06.778238997 +0000 UTC m=+1523.586023216" Nov 02 20:18:06.868190 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:06.868151 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:07.870285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:07.870249 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-q5tqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:18:07.870683 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:07.870302 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 20:18:08.869157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:08.869123 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-q5tqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:18:08.869342 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:08.869175 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 20:18:09.108846 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:09.106086 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:09.375971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:09.375935 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:09.581960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:09.581900 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:09.874613 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:09.874558 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-q5tqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:18:09.874957 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:09.874910 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 20:18:10.868186 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:10.868145 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-q5tqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:18:10.868667 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:10.868204 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 20:18:11.868361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:11.868324 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-q5tqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:18:11.868862 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:11.868392 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 20:18:12.869139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:12.869097 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-q5tqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:18:12.869554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:12.869175 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 20:18:13.752744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:13.752705 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:13.868151 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:13.868110 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-q5tqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:18:13.868382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:13.868204 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 20:18:14.925197 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:14.925158 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:14.925936 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:14.925899 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:14.927104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:14.927080 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:18:18.286246 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:18.286213 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:18.788513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:18.788483 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:22.853787 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:22.853734 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:24.831706 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:24.831623 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:27.467959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:27.463204 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:32.029618 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:32.029555 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:32.219561 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:32.219525 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:33.793470 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:18:33.793425 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:18:36.650240 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:36.650185 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:41.159322 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:41.159281 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:45.534617 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:45.534526 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:45.710855 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:45.710818 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:45.912568 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:45.912540 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:49.074020 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:49.073984 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx"] Nov 02 20:18:49.074420 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:49.074219 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" podUID="2e5d96d4-6d30-4ef3-9a1d-5c384355d899" containerName="istio-proxy" containerID="cri-o://6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474" gracePeriod=30 Nov 02 20:18:49.746231 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:49.746193 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:50.209244 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:50.209205 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:51.407396 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:51.407364 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:51.652697 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:51.652663 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:51.994072 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:51.994037 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:54.378511 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.378478 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:18:54.451067 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.451031 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-envoy\") pod \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " Nov 02 20:18:54.451332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.451311 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-kube-api-access-wzsrj\" (UniqueName: \"kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-kube-api-access-wzsrj\") pod \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " Nov 02 20:18:54.451466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.451450 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-token\" (UniqueName: \"kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-token\") pod \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " Nov 02 20:18:54.451517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.451482 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-credential-socket\") pod \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " Nov 02 20:18:54.451567 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.451520 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-certs\") pod \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " Nov 02 20:18:54.451567 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.451552 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-socket\") pod \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " Nov 02 20:18:54.451663 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.451589 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-podinfo\") pod \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " Nov 02 20:18:54.451663 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.451615 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istiod-ca-cert\") pod \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " Nov 02 20:18:54.451763 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.451674 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-data\") pod \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\" (UID: \"2e5d96d4-6d30-4ef3-9a1d-5c384355d899\") " Nov 02 20:18:54.452195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.451812 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-credential-socket" pod "2e5d96d4-6d30-4ef3-9a1d-5c384355d899" (UID: "2e5d96d4-6d30-4ef3-9a1d-5c384355d899"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:54.452195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.451885 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-certs" pod "2e5d96d4-6d30-4ef3-9a1d-5c384355d899" (UID: "2e5d96d4-6d30-4ef3-9a1d-5c384355d899"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:54.452195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.452093 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-socket" pod "2e5d96d4-6d30-4ef3-9a1d-5c384355d899" (UID: "2e5d96d4-6d30-4ef3-9a1d-5c384355d899"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:54.452195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.452154 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-data" pod "2e5d96d4-6d30-4ef3-9a1d-5c384355d899" (UID: "2e5d96d4-6d30-4ef3-9a1d-5c384355d899"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:54.452195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.452182 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istiod-ca-cert" pod "2e5d96d4-6d30-4ef3-9a1d-5c384355d899" (UID: "2e5d96d4-6d30-4ef3-9a1d-5c384355d899"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:54.452505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.452387 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:54.452505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.452400 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:54.452505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.452409 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:54.452505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.452418 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:54.452505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.452427 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:54.456236 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.456205 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-kube-api-access-wzsrj" pod "2e5d96d4-6d30-4ef3-9a1d-5c384355d899" (UID: "2e5d96d4-6d30-4ef3-9a1d-5c384355d899"). InnerVolumeSpecName "kube-api-access-wzsrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:54.456516 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.456487 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-podinfo" pod "2e5d96d4-6d30-4ef3-9a1d-5c384355d899" (UID: "2e5d96d4-6d30-4ef3-9a1d-5c384355d899"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:18:54.456625 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.456545 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-envoy" pod "2e5d96d4-6d30-4ef3-9a1d-5c384355d899" (UID: "2e5d96d4-6d30-4ef3-9a1d-5c384355d899"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:54.456625 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.456567 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-token" pod "2e5d96d4-6d30-4ef3-9a1d-5c384355d899" (UID: "2e5d96d4-6d30-4ef3-9a1d-5c384355d899"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:54.554296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.554263 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:54.554296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.554296 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:54.554497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.554309 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzsrj\" (UniqueName: \"kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-kube-api-access-wzsrj\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:54.554497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.554323 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e5d96d4-6d30-4ef3-9a1d-5c384355d899-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:54.903092 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.903051 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:54.994175 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.994139 5561 generic.go:356] "Generic (PLEG): container finished" podID="2e5d96d4-6d30-4ef3-9a1d-5c384355d899" containerID="6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474" exitCode=0 Nov 02 20:18:54.994383 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.994203 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" Nov 02 20:18:54.994383 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.994279 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" event={"ID":"2e5d96d4-6d30-4ef3-9a1d-5c384355d899","Type":"ContainerDied","Data":"6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474"} Nov 02 20:18:54.994383 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.994345 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx" event={"ID":"2e5d96d4-6d30-4ef3-9a1d-5c384355d899","Type":"ContainerDied","Data":"1dfca6473eeace1cfd02b6389a51e5f1a0e2eaee8085aba3ec2c4908f927610b"} Nov 02 20:18:54.994383 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:54.994374 5561 scope.go:122] "RemoveContainer" containerID="6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474" Nov 02 20:18:55.010075 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:55.010046 5561 scope.go:122] "RemoveContainer" containerID="6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474" Nov 02 20:18:55.010461 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:18:55.010437 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474\": container with ID starting with 6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474 not found: ID does not exist" containerID="6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474" Nov 02 20:18:55.010553 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:55.010475 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474"} err="failed to get container status \"6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474\": rpc error: code = NotFound desc = could not find container \"6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474\": container with ID starting with 6a31acf2b5a83fa0e7ff4be9d630056c285ffba278473876c76dae431f983474 not found: ID does not exist" Nov 02 20:18:55.152698 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:55.152660 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx"] Nov 02 20:18:55.200461 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:55.200371 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-24dxx"] Nov 02 20:18:55.314447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:55.314414 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5d96d4-6d30-4ef3-9a1d-5c384355d899" path="/var/lib/kubelet/pods/2e5d96d4-6d30-4ef3-9a1d-5c384355d899/volumes" Nov 02 20:18:57.629227 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:57.629191 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:57.896943 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:57.896853 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:59.093516 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.093482 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch"] Nov 02 20:18:59.094094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.094072 5561 cpu_manager.go:414] "containerMap: removing container" podUID="2e5d96d4-6d30-4ef3-9a1d-5c384355d899" containerName="istio-proxy" Nov 02 20:18:59.094094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.094087 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5d96d4-6d30-4ef3-9a1d-5c384355d899" containerName="istio-proxy" Nov 02 20:18:59.094251 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.094213 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5d96d4-6d30-4ef3-9a1d-5c384355d899" containerName="istio-proxy" Nov 02 20:18:59.099163 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.099141 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.150687 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.150653 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch"] Nov 02 20:18:59.194283 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.194254 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-be1348-serving-cert\"" Nov 02 20:18:59.219538 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.219511 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.219703 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.219654 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjxw\" (UniqueName: \"kubernetes.io/projected/ec07656a-a444-401f-81f1-b229a22f4e13-kube-api-access-vmjxw\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.219787 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.219770 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec07656a-a444-401f-81f1-b229a22f4e13-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.219829 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.219813 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.321018 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.320978 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.321192 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:18:59.321096 5561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-serving-cert: secret "maas-api-e2e-shared-a-w1-be1348-serving-cert" not found Nov 02 20:18:59.321192 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.321135 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjxw\" (UniqueName: \"kubernetes.io/projected/ec07656a-a444-401f-81f1-b229a22f4e13-kube-api-access-vmjxw\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.321192 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:18:59.321166 5561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-maas-api-tls podName:ec07656a-a444-401f-81f1-b229a22f4e13 nodeName:}" failed. No retries permitted until 2026-11-02 20:18:59.821149158 +0000 UTC m=+1576.628933353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-maas-api-tls") pod "maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" (UID: "ec07656a-a444-401f-81f1-b229a22f4e13") : secret "maas-api-e2e-shared-a-w1-be1348-serving-cert" not found Nov 02 20:18:59.321381 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.321306 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec07656a-a444-401f-81f1-b229a22f4e13-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.321442 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.321381 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.322192 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.322170 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec07656a-a444-401f-81f1-b229a22f4e13-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.323963 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.323942 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.348381 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.348304 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:59.378211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.378169 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjxw\" (UniqueName: \"kubernetes.io/projected/ec07656a-a444-401f-81f1-b229a22f4e13-kube-api-access-vmjxw\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.519989 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.519946 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:59.691365 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.691322 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:59.828735 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.828702 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:18:59.831239 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:18:59.831217 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:19:00.013750 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:00.013667 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:19:00.391419 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:00.391339 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch"] Nov 02 20:19:00.397866 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:19:00.397838 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec07656a_a444_401f_81f1_b229a22f4e13.slice/crio-52c488f75b027d49656ee94bda5ae1b9b2788dbafc69c7a1c5a8afc3ced52447 WatchSource:0}: Error finding container 52c488f75b027d49656ee94bda5ae1b9b2788dbafc69c7a1c5a8afc3ced52447: Status 404 returned error can't find the container with id 52c488f75b027d49656ee94bda5ae1b9b2788dbafc69c7a1c5a8afc3ced52447 Nov 02 20:19:00.399270 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:00.399254 5561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:19:01.037795 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:01.037759 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" event={"ID":"ec07656a-a444-401f-81f1-b229a22f4e13","Type":"ContainerStarted","Data":"52c488f75b027d49656ee94bda5ae1b9b2788dbafc69c7a1c5a8afc3ced52447"} Nov 02 20:19:02.047735 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:02.047697 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" event={"ID":"ec07656a-a444-401f-81f1-b229a22f4e13","Type":"ContainerStarted","Data":"ae2a71ea411b05fefcae16cbd3bab47c191e80712e48e35b62c58985f3485a24"} Nov 02 20:19:02.048185 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:02.047900 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:19:02.157426 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:02.157362 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" podStartSLOduration=2.316755804 podStartE2EDuration="3.157342405s" totalImagesPullingTime="840.586601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:02.15593968 +0000 UTC m=+1578.963723917" watchObservedRunningTime="2026-11-02 20:19:02.157342405 +0000 UTC m=+1578.965126623" Nov 02 20:19:03.165389 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:03.165353 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:03.770130 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:03.770046 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:03.812069 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:19:03.812025 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:19:04.063292 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:04.063213 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:04.634988 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:04.634949 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:08.060390 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:08.060360 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:19:08.908600 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:08.908548 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:09.100065 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:09.100032 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:12.679686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:12.679649 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:12.854966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:12.854921 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:14.235690 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:14.235451 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:15.940995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:15.940952 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-q5tqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:19:15.942060 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:15.942023 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:19:17.148330 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:17.148263 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:17.325712 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:17.325677 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:18.977707 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:18.977666 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:23.462589 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:23.462535 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:28.080988 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:28.080955 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:32.678274 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:32.678234 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:33.780723 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:19:33.780690 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:19:37.701017 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:37.700934 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:42.614144 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:42.614087 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:47.377427 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:47.377395 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:49.015171 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:49.015131 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:49.199845 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:49.199811 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:50.098581 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:50.098543 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:50.348927 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:50.348822 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:50.524383 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:50.524349 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:52.348091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:52.348053 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:57.490720 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:19:57.490641 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:02.317147 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:02.317112 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:03.546211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:03.546179 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:03.780796 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:20:03.780760 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:20:04.028636 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:04.028601 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:04.831934 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:04.831883 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:05.007996 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:05.007959 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:05.239166 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:05.239122 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:05.656261 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:05.656221 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:07.031088 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:07.031030 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:07.471450 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:07.471385 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:08.023861 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:08.023830 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:10.043992 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:10.043947 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:10.231576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:10.231546 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:11.572780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:11.572739 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:12.233668 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:12.233598 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:13.479944 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:13.479870 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:13.669539 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:13.669473 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:13.967809 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:13.967769 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:14.314242 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:14.314150 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:14.489070 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:14.488991 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:14.758043 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:14.758004 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:15.840667 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:15.840632 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:16.020103 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:16.020063 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:18.500457 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:18.500420 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:20.650338 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:20.650302 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:20.849554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:20.849508 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:21.028168 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:21.027994 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:21.470116 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:21.470083 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:21.778164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:21.778082 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:21.983184 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:21.983127 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:22.386818 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:22.386780 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:23.170172 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:23.170136 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:27.184779 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:27.184738 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:28.150488 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:28.150446 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:28.339066 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:28.339023 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:28.692181 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:28.692149 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:28.889713 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:28.889678 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:29.074862 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:29.073185 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:30.670917 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:30.670852 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:31.466628 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:31.466592 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:32.099825 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:32.099783 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:32.442002 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:32.441951 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:33.787147 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:20:33.787113 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:20:33.802045 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:33.802010 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:36.239986 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:36.239945 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:36.426376 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:36.426345 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:36.609446 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:36.609362 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:38.068713 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:38.068676 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:39.691116 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:39.691082 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:39.973391 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:39.973299 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:40.150913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:40.150842 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:40.826534 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:40.826498 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:41.607600 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:41.607569 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:42.184193 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:42.184108 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:42.943633 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:42.943599 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:44.962689 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:44.962651 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:47.163203 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:47.163169 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:47.349571 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:47.349537 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:49.792463 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:49.792428 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:50.409280 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:50.409244 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:50.584168 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:50.584135 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:50.757719 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:50.757629 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:54.849171 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:54.849126 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:59.464211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:20:59.464173 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:01.350733 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:01.350695 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:03.779457 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:21:03.779419 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:21:03.911214 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:03.911179 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:04.092599 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:04.092496 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:06.014257 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:06.012768 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:06.352236 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:06.352151 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt"] Nov 02 20:21:06.352376 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:06.352334 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" containerID="cri-o://b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309" gracePeriod=30 Nov 02 20:21:07.514322 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:07.514283 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:09.043249 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:09.043210 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:10.512015 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:10.511982 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:11.677448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.677413 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:21:11.760272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760240 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-envoy\") pod \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " Nov 02 20:21:11.760272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760278 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-credential-socket\") pod \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " Nov 02 20:21:11.760497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760308 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-certs\") pod \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " Nov 02 20:21:11.760497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760357 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-podinfo\") pod \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " Nov 02 20:21:11.760497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760384 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-kube-api-access-ztkrw\" (UniqueName: \"kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-kube-api-access-ztkrw\") pod \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " Nov 02 20:21:11.760497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760465 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istiod-ca-cert\") pod \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " Nov 02 20:21:11.760497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760495 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-socket\") pod \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " Nov 02 20:21:11.760709 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760535 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-token\" (UniqueName: \"kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-token\") pod \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " Nov 02 20:21:11.760709 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760583 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-data\") pod \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\" (UID: \"7a341b73-b2fd-46f6-b81a-8b7bf2f06355\") " Nov 02 20:21:11.760709 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760590 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-credential-socket" pod "7a341b73-b2fd-46f6-b81a-8b7bf2f06355" (UID: "7a341b73-b2fd-46f6-b81a-8b7bf2f06355"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.760709 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760628 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-certs" pod "7a341b73-b2fd-46f6-b81a-8b7bf2f06355" (UID: "7a341b73-b2fd-46f6-b81a-8b7bf2f06355"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.760968 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760937 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-socket" pod "7a341b73-b2fd-46f6-b81a-8b7bf2f06355" (UID: "7a341b73-b2fd-46f6-b81a-8b7bf2f06355"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.761046 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.760965 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-data" pod "7a341b73-b2fd-46f6-b81a-8b7bf2f06355" (UID: "7a341b73-b2fd-46f6-b81a-8b7bf2f06355"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.761219 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.761162 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istiod-ca-cert" pod "7a341b73-b2fd-46f6-b81a-8b7bf2f06355" (UID: "7a341b73-b2fd-46f6-b81a-8b7bf2f06355"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:11.761397 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.761375 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.761453 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.761407 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.761453 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.761424 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.761453 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.761439 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.761563 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.761453 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.762622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.762601 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-envoy" pod "7a341b73-b2fd-46f6-b81a-8b7bf2f06355" (UID: "7a341b73-b2fd-46f6-b81a-8b7bf2f06355"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.762826 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.762808 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-token" pod "7a341b73-b2fd-46f6-b81a-8b7bf2f06355" (UID: "7a341b73-b2fd-46f6-b81a-8b7bf2f06355"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:11.763091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.763071 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-podinfo" pod "7a341b73-b2fd-46f6-b81a-8b7bf2f06355" (UID: "7a341b73-b2fd-46f6-b81a-8b7bf2f06355"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:21:11.763091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.763076 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-kube-api-access-ztkrw" pod "7a341b73-b2fd-46f6-b81a-8b7bf2f06355" (UID: "7a341b73-b2fd-46f6-b81a-8b7bf2f06355"). InnerVolumeSpecName "kube-api-access-ztkrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:11.863177 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.863139 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.863177 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.863171 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.863377 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.863186 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztkrw\" (UniqueName: \"kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-kube-api-access-ztkrw\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.863377 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:11.863201 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a341b73-b2fd-46f6-b81a-8b7bf2f06355-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:12.049686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:12.049646 5561 generic.go:356] "Generic (PLEG): container finished" podID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerID="b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309" exitCode=0 Nov 02 20:21:12.049686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:12.049696 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" Nov 02 20:21:12.049995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:12.049701 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" event={"ID":"7a341b73-b2fd-46f6-b81a-8b7bf2f06355","Type":"ContainerDied","Data":"b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309"} Nov 02 20:21:12.049995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:12.049766 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt" event={"ID":"7a341b73-b2fd-46f6-b81a-8b7bf2f06355","Type":"ContainerDied","Data":"a6604d799f7d1a117c2022207f01e03997651c5fd913f02bb2e31b376e7fe42c"} Nov 02 20:21:12.049995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:12.049807 5561 scope.go:122] "RemoveContainer" containerID="b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309" Nov 02 20:21:12.064161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:12.064136 5561 scope.go:122] "RemoveContainer" containerID="b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309" Nov 02 20:21:12.064602 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:21:12.064578 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309\": container with ID starting with b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309 not found: ID does not exist" containerID="b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309" Nov 02 20:21:12.064705 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:12.064610 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309"} err="failed to get container status \"b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309\": rpc error: code = NotFound desc = could not find container \"b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309\": container with ID starting with b1900d075fb8db2aa89fad1f26225800c35613d32c16d9c26f302b95972c9309 not found: ID does not exist" Nov 02 20:21:12.217772 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:12.217700 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt"] Nov 02 20:21:12.222010 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:12.221561 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-wlrqt"] Nov 02 20:21:13.314339 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:13.314305 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" path="/var/lib/kubelet/pods/7a341b73-b2fd-46f6-b81a-8b7bf2f06355/volumes" Nov 02 20:21:13.866411 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:13.866334 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:18.366901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:18.364829 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:21.138247 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:21.138205 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd"] Nov 02 20:21:21.138647 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:21.138501 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" podUID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" containerName="istio-proxy" containerID="cri-o://10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b" gracePeriod=30 Nov 02 20:21:21.183855 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:21.183811 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz"] Nov 02 20:21:21.184095 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:21.184065 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" podUID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" containerName="istio-proxy" containerID="cri-o://f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6" gracePeriod=30 Nov 02 20:21:22.957898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:22.957082 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:26.456272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.456233 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:21:26.501644 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.501622 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:21:26.581373 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.581336 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-data\") pod \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " Nov 02 20:21:26.581542 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.581403 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-socket\") pod \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " Nov 02 20:21:26.581542 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.581443 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-podinfo\") pod \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " Nov 02 20:21:26.581542 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.581475 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-token\" (UniqueName: \"kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-token\") pod \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " Nov 02 20:21:26.581681 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.581554 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-kube-api-access-st982\" (UniqueName: \"kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-kube-api-access-st982\") pod \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " Nov 02 20:21:26.581681 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.581646 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-credential-socket\") pod \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " Nov 02 20:21:26.581808 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.581671 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-data" pod "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" (UID: "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:26.581808 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.581683 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istiod-ca-cert\") pod \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " Nov 02 20:21:26.581808 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.581708 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-socket" pod "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" (UID: "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:26.581808 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.581787 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-certs\") pod \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " Nov 02 20:21:26.582074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.581919 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-envoy\") pod \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\" (UID: \"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c\") " Nov 02 20:21:26.582074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.581953 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-credential-socket" pod "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" (UID: "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:26.582074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.582052 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istiod-ca-cert" pod "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" (UID: "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:26.582074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.582057 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-certs" pod "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" (UID: "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:26.582725 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.582702 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.582796 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.582732 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.582796 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.582749 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.582796 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.582767 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.582796 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.582783 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.583938 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.583915 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-kube-api-access-st982" pod "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" (UID: "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c"). InnerVolumeSpecName "kube-api-access-st982". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:26.584233 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.584213 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-envoy" pod "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" (UID: "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:26.584311 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.584292 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-podinfo" pod "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" (UID: "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:21:26.584349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.584323 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-token" pod "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" (UID: "68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:26.683628 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.683555 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-credential-socket\") pod \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " Nov 02 20:21:26.683771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.683683 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-certs\") pod \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " Nov 02 20:21:26.683771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.683713 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-data\") pod \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " Nov 02 20:21:26.683771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.683754 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-token\" (UniqueName: \"kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-token\") pod \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " Nov 02 20:21:26.683905 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.683796 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-envoy\") pod \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " Nov 02 20:21:26.683905 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.683828 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-credential-socket" pod "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" (UID: "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:26.684051 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684030 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-data" pod "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" (UID: "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:26.684051 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684038 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-socket\") pod \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " Nov 02 20:21:26.684139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684041 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-certs" pod "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" (UID: "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:26.684139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684101 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istiod-ca-cert\") pod \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " Nov 02 20:21:26.684231 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684170 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-podinfo\") pod \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " Nov 02 20:21:26.684231 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684217 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-socket" pod "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" (UID: "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:26.684346 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684268 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-kube-api-access-j6h2z\" (UniqueName: \"kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-kube-api-access-j6h2z\") pod \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\" (UID: \"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b\") " Nov 02 20:21:26.684529 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684477 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istiod-ca-cert" pod "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" (UID: "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:26.684845 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684827 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.684952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684851 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.684952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684864 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.684952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684908 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st982\" (UniqueName: \"kubernetes.io/projected/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-kube-api-access-st982\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.684952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684924 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.684952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684937 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.684952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684950 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.685152 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684963 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.685152 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.684976 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.686123 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.686099 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-token" pod "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" (UID: "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:26.686324 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.686302 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-kube-api-access-j6h2z" pod "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" (UID: "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b"). InnerVolumeSpecName "kube-api-access-j6h2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:26.686454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.686436 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-podinfo" pod "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" (UID: "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:21:26.686491 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.686440 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-envoy" pod "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" (UID: "7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:26.788773 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.788736 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.788773 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.788769 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6h2z\" (UniqueName: \"kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-kube-api-access-j6h2z\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.788773 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.788780 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.789027 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:26.788789 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:27.154064 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.154031 5561 generic.go:356] "Generic (PLEG): container finished" podID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" containerID="f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6" exitCode=0 Nov 02 20:21:27.154262 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.154095 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" Nov 02 20:21:27.154262 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.154122 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" event={"ID":"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b","Type":"ContainerDied","Data":"f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6"} Nov 02 20:21:27.154262 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.154212 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz" event={"ID":"7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b","Type":"ContainerDied","Data":"fcfb888161a239b7e3a34d8d1b89e4811443fa7b91fadd44317b45a936e92160"} Nov 02 20:21:27.154262 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.154240 5561 scope.go:122] "RemoveContainer" containerID="f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6" Nov 02 20:21:27.155742 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.155718 5561 generic.go:356] "Generic (PLEG): container finished" podID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" containerID="10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b" exitCode=0 Nov 02 20:21:27.155851 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.155746 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" event={"ID":"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c","Type":"ContainerDied","Data":"10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b"} Nov 02 20:21:27.155851 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.155798 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" Nov 02 20:21:27.156031 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.155797 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd" event={"ID":"68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c","Type":"ContainerDied","Data":"3a42ae9f807a83dad63e14e9ecaba435f8da737096f906d81a317c3bdbcfcbcf"} Nov 02 20:21:27.177427 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.177404 5561 scope.go:122] "RemoveContainer" containerID="f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6" Nov 02 20:21:27.177705 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:21:27.177686 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6\": container with ID starting with f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6 not found: ID does not exist" containerID="f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6" Nov 02 20:21:27.177759 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.177714 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6"} err="failed to get container status \"f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6\": rpc error: code = NotFound desc = could not find container \"f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6\": container with ID starting with f1ed27db634977745f532f7e1651c4c88d0faa0a481be9346c05e44e50a235c6 not found: ID does not exist" Nov 02 20:21:27.177759 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.177731 5561 scope.go:122] "RemoveContainer" containerID="10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b" Nov 02 20:21:27.186961 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.186943 5561 scope.go:122] "RemoveContainer" containerID="10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b" Nov 02 20:21:27.187215 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:21:27.187196 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b\": container with ID starting with 10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b not found: ID does not exist" containerID="10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b" Nov 02 20:21:27.187264 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.187222 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b"} err="failed to get container status \"10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b\": rpc error: code = NotFound desc = could not find container \"10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b\": container with ID starting with 10bc83aa93043ca3e79ca3f51d31ea3f267e500c3246db9c2ec0041fb7659a0b not found: ID does not exist" Nov 02 20:21:27.315149 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.315114 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd"] Nov 02 20:21:27.358527 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.358492 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-qxxkd"] Nov 02 20:21:27.417716 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.417633 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz"] Nov 02 20:21:27.421036 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.421014 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-w4mmz"] Nov 02 20:21:27.444033 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:27.443992 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:29.313945 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:29.313914 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" path="/var/lib/kubelet/pods/68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c/volumes" Nov 02 20:21:29.314406 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:29.314393 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" path="/var/lib/kubelet/pods/7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b/volumes" Nov 02 20:21:31.834410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:31.834376 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:32.014422 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:32.014391 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:32.198642 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:32.198604 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:33.781132 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:21:33.781088 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:21:36.232586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:36.232536 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:38.733409 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:38.733374 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:41.276621 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:41.276560 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:45.714243 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:45.714207 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:45.891194 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:45.891151 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:46.067860 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:46.067743 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:50.398916 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:50.398862 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:50.807469 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:50.807426 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:54.950410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:54.950368 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:55.394594 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:55.394515 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:59.462259 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:21:59.462143 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:01.024368 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:01.024331 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:03.780527 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:22:03.780488 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:22:03.918610 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:03.917232 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:08.479970 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:08.479927 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:12.598699 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:12.598646 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:13.783771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:13.783735 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:18.374345 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:18.374282 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:23.330516 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:23.330440 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:27.868737 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:27.868613 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:29.161826 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:29.161784 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:29.769704 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:29.769636 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:30.156029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:30.155998 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:31.871769 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:31.871732 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:32.367721 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:32.367682 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:32.546899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:32.545599 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:32.935351 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:32.935317 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:33.781193 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:22:33.781156 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:22:34.191424 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:34.191391 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:34.381171 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:34.381136 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:37.351012 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:37.350974 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:37.571039 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:37.571000 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:37.736692 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:37.736656 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:38.521749 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:38.521709 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:38.714807 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:38.714771 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:38.900316 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:38.900234 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:39.205892 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:39.205838 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:39.737335 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:39.737255 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:39.937420 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:39.937282 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:42.308282 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:42.308243 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:44.155934 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:44.155867 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:44.355689 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:44.355613 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:45.476092 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:45.476047 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:45.765412 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:45.765327 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:47.300745 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:47.300697 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:47.491688 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:47.491645 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:48.184421 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:48.184389 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:48.361789 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:48.361750 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:51.163690 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:51.163654 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:51.564009 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:51.563844 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:52.342537 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:52.342502 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:54.540818 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:54.540720 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:54.725585 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:54.725546 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:54.966434 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:54.966401 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:55.594237 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:55.594195 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:56.399364 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:56.399326 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:57.500415 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:57.500370 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:59.852342 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:22:59.852310 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:00.039692 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:00.039652 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:02.563652 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:02.563621 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:03.141226 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:03.141190 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:03.318155 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:03.318122 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:03.815143 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:23:03.815103 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:23:07.190519 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:07.190394 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:11.690905 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:11.687955 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:16.141085 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:16.141043 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:20.622344 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:20.621614 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:23.143492 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:23.143454 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:23.316260 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:23.316230 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:25.172865 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:25.172821 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:30.765958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:30.765915 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:33.779268 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:23:33.779227 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:23:34.139018 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:34.138939 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:34.355946 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:34.351704 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:34.524497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:34.524463 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:36.557308 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:36.557274 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:38.803365 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:38.803323 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:42.349118 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:42.349084 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:43.232172 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:43.232130 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:43.407708 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:43.407662 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:43.605173 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:43.605094 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:48.115455 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:48.115408 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:51.515506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:51.515466 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:53.040794 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:53.040742 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:57.604566 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:57.604525 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:57.803273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:57.803242 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:57.978357 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:57.978316 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:59.472338 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:23:59.472287 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:00.188386 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:00.188320 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:00.365029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:00.364970 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:02.288866 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:02.288823 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:03.779117 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:24:03.779081 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:24:06.749965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:06.749910 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:11.360014 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:11.359974 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:15.516556 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.516523 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9"] Nov 02 20:24:15.517161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.517122 5561 cpu_manager.go:414] "containerMap: removing container" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" Nov 02 20:24:15.517161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.517134 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" Nov 02 20:24:15.517161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.517147 5561 cpu_manager.go:414] "containerMap: removing container" podUID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" containerName="istio-proxy" Nov 02 20:24:15.517161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.517152 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" containerName="istio-proxy" Nov 02 20:24:15.517161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.517161 5561 cpu_manager.go:414] "containerMap: removing container" podUID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" containerName="istio-proxy" Nov 02 20:24:15.517332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.517166 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" containerName="istio-proxy" Nov 02 20:24:15.519923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.518007 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae3c417-3c1b-4e3f-8ed8-ce611c9c6c0b" containerName="istio-proxy" Nov 02 20:24:15.519923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.518072 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a341b73-b2fd-46f6-b81a-8b7bf2f06355" containerName="istio-proxy" Nov 02 20:24:15.519923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.518091 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="68db2292-dd9e-4dc3-b0e3-fb2a0d3b2b8c" containerName="istio-proxy" Nov 02 20:24:15.523541 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.523519 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.573710 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.573670 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7dcb03-openshift-default-dockercfg-mr2wr\"" Nov 02 20:24:15.693377 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.693331 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.693571 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.693472 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.693642 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.693622 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c14fb-7305-431d-ad95-e51e4ea1b417-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.693700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.693669 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.693761 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.693736 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.693799 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.693789 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.693834 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.693810 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tq7v\" (UniqueName: \"kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-kube-api-access-7tq7v\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.694019 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.693998 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.694102 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.694039 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.724555 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.724521 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9"] Nov 02 20:24:15.762491 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.762438 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:15.795240 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.795148 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.795240 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.795198 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.795472 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.795283 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.795472 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.795340 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.795472 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.795422 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c14fb-7305-431d-ad95-e51e4ea1b417-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.795472 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.795468 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.795682 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.795498 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.795682 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.795534 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.795682 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.795566 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tq7v\" (UniqueName: \"kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-kube-api-access-7tq7v\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.795860 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.795817 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.795860 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.795849 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.796044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.796014 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.796187 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.796166 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.796305 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.796286 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c14fb-7305-431d-ad95-e51e4ea1b417-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.797998 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.797972 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.798172 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.798154 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.849337 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.849289 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.849522 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.849405 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tq7v\" (UniqueName: \"kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-kube-api-access-7tq7v\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:15.975441 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:15.975404 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:16.142863 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:16.142782 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:16.227497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:16.227421 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:16.582466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:16.582429 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9"] Nov 02 20:24:16.631562 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:24:16.631522 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29c14fb_7305_431d_ad95_e51e4ea1b417.slice/crio-9639379c5339b967f15d211b405605bee8e5e1f79347d94bac5eb4f81eb78fbd WatchSource:0}: Error finding container 9639379c5339b967f15d211b405605bee8e5e1f79347d94bac5eb4f81eb78fbd: Status 404 returned error can't find the container with id 9639379c5339b967f15d211b405605bee8e5e1f79347d94bac5eb4f81eb78fbd Nov 02 20:24:16.634262 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:16.634227 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:24:16.634373 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:16.634312 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:24:16.634373 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:16.634360 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:24:17.385576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:17.385534 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" event={"ID":"c29c14fb-7305-431d-ad95-e51e4ea1b417","Type":"ContainerStarted","Data":"33d492eb5403cc48f9d844a19627396134dd6cad4f1f0b12e16db2b2f272e481"} Nov 02 20:24:17.385803 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:17.385591 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" event={"ID":"c29c14fb-7305-431d-ad95-e51e4ea1b417","Type":"ContainerStarted","Data":"9639379c5339b967f15d211b405605bee8e5e1f79347d94bac5eb4f81eb78fbd"} Nov 02 20:24:17.606723 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:17.606671 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" podStartSLOduration=2.606655856 podStartE2EDuration="2.606655856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:17.557437582 +0000 UTC m=+1894.365221798" watchObservedRunningTime="2026-11-02 20:24:17.606655856 +0000 UTC m=+1894.414440071" Nov 02 20:24:18.144901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.143892 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:18.153899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.149998 5561 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:24:18.153899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.150060 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" podUID="c29c14fb-7305-431d-ad95-e51e4ea1b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 20:24:18.153899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.151075 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5"] Nov 02 20:24:18.156860 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.156835 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.229102 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.229070 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.229452 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.229424 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.229614 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.229598 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.229820 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.229801 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z88vw\" (UniqueName: \"kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-kube-api-access-z88vw\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.230000 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.229983 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.230109 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.230091 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.230244 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.230229 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.230407 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.230390 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.230622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.230605 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.274925 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.274855 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5"] Nov 02 20:24:18.277472 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.277446 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c76017-openshift-default-dockercfg-p9q56\"" Nov 02 20:24:18.334768 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.334728 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.335537 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.335514 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.344998 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.344970 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.345280 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.345259 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z88vw\" (UniqueName: \"kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-kube-api-access-z88vw\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.352905 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.345445 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.353190 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.348357 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.353190 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.335417 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.353190 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.349423 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.353753 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.353100 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.353943 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.353923 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.357028 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.356994 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.367921 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.358605 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.368262 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.357730 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.368365 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.367375 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.368667 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.368646 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.369148 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.369123 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.403036 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.403006 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z88vw\" (UniqueName: \"kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-kube-api-access-z88vw\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.410545 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.410517 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:18.478285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:18.478253 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:19.026906 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:19.026815 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5"] Nov 02 20:24:19.058324 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:24:19.058292 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd90ccb4_69c5_498b_b5d9_bdb093974c60.slice/crio-4e626518221d364306006dbd76d4248d83790b8192988bf07c515307d75187ef WatchSource:0}: Error finding container 4e626518221d364306006dbd76d4248d83790b8192988bf07c515307d75187ef: Status 404 returned error can't find the container with id 4e626518221d364306006dbd76d4248d83790b8192988bf07c515307d75187ef Nov 02 20:24:19.061030 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:19.060995 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:24:19.061117 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:19.061068 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:24:19.061117 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:19.061100 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:24:19.153898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:19.147309 5561 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:24:19.153898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:19.147373 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" podUID="c29c14fb-7305-431d-ad95-e51e4ea1b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 20:24:19.402515 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:19.402479 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" event={"ID":"fd90ccb4-69c5-498b-b5d9-bdb093974c60","Type":"ContainerStarted","Data":"1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a"} Nov 02 20:24:19.402707 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:19.402545 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" event={"ID":"fd90ccb4-69c5-498b-b5d9-bdb093974c60","Type":"ContainerStarted","Data":"4e626518221d364306006dbd76d4248d83790b8192988bf07c515307d75187ef"} Nov 02 20:24:19.478376 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:19.478338 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:19.515862 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:19.515792 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" podStartSLOduration=1.515778938 podStartE2EDuration="1.515778938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:19.51379956 +0000 UTC m=+1896.321583800" watchObservedRunningTime="2026-11-02 20:24:19.515778938 +0000 UTC m=+1896.323563153" Nov 02 20:24:20.150901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:20.150318 5561 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:24:20.150901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:20.150388 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" podUID="c29c14fb-7305-431d-ad95-e51e4ea1b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 20:24:20.481151 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:20.481109 5561 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:24:20.481345 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:20.481170 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" podUID="fd90ccb4-69c5-498b-b5d9-bdb093974c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 20:24:21.083001 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:21.082950 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:21.153277 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:21.153236 5561 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:24:21.153798 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:21.153298 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" podUID="c29c14fb-7305-431d-ad95-e51e4ea1b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 20:24:21.479382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:21.479347 5561 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:24:21.479603 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:21.479397 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" podUID="fd90ccb4-69c5-498b-b5d9-bdb093974c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 20:24:22.145094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:22.145058 5561 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:24:22.145393 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:22.145361 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" podUID="c29c14fb-7305-431d-ad95-e51e4ea1b417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 20:24:22.478898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:22.478783 5561 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:24:22.479390 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:22.478856 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" podUID="fd90ccb4-69c5-498b-b5d9-bdb093974c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 20:24:23.160585 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:23.160545 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:23.161025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:23.160997 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:23.161948 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:23.161928 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:24:23.480918 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:23.480811 5561 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:24:23.480918 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:23.480866 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" podUID="fd90ccb4-69c5-498b-b5d9-bdb093974c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 20:24:24.478744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:24.478705 5561 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:24:24.478967 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:24.478772 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" podUID="fd90ccb4-69c5-498b-b5d9-bdb093974c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 20:24:25.483294 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:25.483263 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:25.483717 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:25.483696 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:25.484437 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:25.484416 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:24:25.792818 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:25.792741 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:30.288746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:30.288710 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:33.780724 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:24:33.780688 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:24:34.731539 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:34.731498 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:39.257480 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:39.257439 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:43.774145 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:43.774105 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:45.742413 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:45.742379 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84"] Nov 02 20:24:45.746686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:45.746664 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:45.907239 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:45.907200 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:45.907418 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:45.907361 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:45.907418 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:45.907398 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f544e1a-31c2-4601-a0d3-95f214fa213d-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:45.907565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:45.907424 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxkb\" (UniqueName: \"kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-kube-api-access-4rxkb\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:45.907623 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:45.907565 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:45.907712 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:45.907686 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:45.907899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:45.907851 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:45.908054 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:45.908020 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:45.908119 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:45.908063 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:45.937415 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:45.937385 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84"] Nov 02 20:24:46.009252 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.009168 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.009425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.009272 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.009425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.009303 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.009425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.009402 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.009582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.009428 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.009582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.009457 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.009582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.009535 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.009582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.009558 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f544e1a-31c2-4601-a0d3-95f214fa213d-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.009582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.009573 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxkb\" (UniqueName: \"kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-kube-api-access-4rxkb\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.009937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.009622 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.009937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.009680 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.010052 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.009964 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.010276 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.010249 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.010510 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.010493 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f544e1a-31c2-4601-a0d3-95f214fa213d-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.012430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.012404 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.012641 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.012616 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.065663 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.065622 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.065812 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.065736 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxkb\" (UniqueName: \"kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-kube-api-access-4rxkb\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.360945 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.360816 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:46.793068 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.793023 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84"] Nov 02 20:24:46.799812 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:24:46.799778 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f544e1a_31c2_4601_a0d3_95f214fa213d.slice/crio-16508a965bcdf505d1e2b983a7820047eaad457d0909a86a29c77d114dce7a34 WatchSource:0}: Error finding container 16508a965bcdf505d1e2b983a7820047eaad457d0909a86a29c77d114dce7a34: Status 404 returned error can't find the container with id 16508a965bcdf505d1e2b983a7820047eaad457d0909a86a29c77d114dce7a34 Nov 02 20:24:46.802666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.802627 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:24:46.802757 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.802702 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:24:46.802757 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:46.802733 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:24:47.601032 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:47.600999 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" event={"ID":"8f544e1a-31c2-4601-a0d3-95f214fa213d","Type":"ContainerStarted","Data":"49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1"} Nov 02 20:24:47.601217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:47.601051 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" event={"ID":"8f544e1a-31c2-4601-a0d3-95f214fa213d","Type":"ContainerStarted","Data":"16508a965bcdf505d1e2b983a7820047eaad457d0909a86a29c77d114dce7a34"} Nov 02 20:24:47.759227 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:47.759180 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" podStartSLOduration=2.759166594 podStartE2EDuration="2.759166594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:47.758935711 +0000 UTC m=+1924.566719946" watchObservedRunningTime="2026-11-02 20:24:47.759166594 +0000 UTC m=+1924.566950810" Nov 02 20:24:48.309326 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:48.309284 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:48.365911 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:48.365218 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:48.369903 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:48.369443 5561 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:24:48.369903 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:48.369503 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" podUID="8f544e1a-31c2-4601-a0d3-95f214fa213d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 20:24:49.362085 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:49.362047 5561 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:24:49.362489 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:49.362103 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" podUID="8f544e1a-31c2-4601-a0d3-95f214fa213d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 20:24:50.361369 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:50.361335 5561 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:24:50.361602 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:50.361404 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" podUID="8f544e1a-31c2-4601-a0d3-95f214fa213d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 20:24:51.361605 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:51.361569 5561 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:24:51.362023 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:51.361625 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" podUID="8f544e1a-31c2-4601-a0d3-95f214fa213d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 20:24:51.873779 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:51.873732 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:52.269474 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:52.269436 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:52.371901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:52.368256 5561 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:24:52.371901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:52.368318 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" podUID="8f544e1a-31c2-4601-a0d3-95f214fa213d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 20:24:52.459923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:52.459870 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:53.367184 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:53.367156 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:53.367666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:53.367648 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:53.368627 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:53.368606 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:24:53.464415 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:53.464365 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:53.650035 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:53.649911 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:53.849694 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:53.849660 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:57.201108 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:57.201067 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:57.624630 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:57.624468 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:57.812917 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:57.812865 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:58.013440 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:58.013401 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:58.399951 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:58.377817 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:58.568418 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:58.568372 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:58.827220 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:24:58.827185 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:01.711676 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:01.711639 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:01.891360 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:01.891326 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:03.536316 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:03.536275 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:03.796885 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:25:03.796791 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:25:03.894135 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:03.894100 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:04.106082 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:04.105992 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:05.299766 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:05.299660 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:06.102477 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:06.102449 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:07.084901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:07.080626 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:07.314332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:07.314295 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:07.735677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:07.727658 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:08.591552 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:08.591510 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:09.406677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:09.406644 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:10.377252 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:10.377218 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:11.453929 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:11.453860 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:11.756758 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:11.756671 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:13.415282 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:13.415234 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:14.715225 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:14.715195 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:14.906966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:14.906897 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:15.090732 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:15.090654 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:15.279147 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:15.279114 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:19.004855 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:19.004816 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:19.184483 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:19.184451 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:22.573284 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:22.573243 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:23.318083 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:23.318044 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:23.708316 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:23.708286 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:25.062685 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:25.062632 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:25.941230 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:25.941192 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:28.231968 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:28.231924 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:28.477195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:28.477156 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:28.648208 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:28.648119 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:29.939083 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:29.939047 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:30.117537 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:30.117508 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:33.271760 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:33.271719 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:33.779454 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:25:33.779417 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:25:37.645675 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:37.645641 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p"] Nov 02 20:25:37.646196 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:37.645856 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" containerID="cri-o://709a4646e23453c1ea631b0dfbfbbe5ac188355a47adaea53a319cb0b1bc07f0" gracePeriod=30 Nov 02 20:25:37.693087 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:37.693053 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6"] Nov 02 20:25:37.693349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:37.693301 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" containerID="cri-o://f5df2de9ae8ae1df2d32c983e0670cd2e5fdd26b3982fb77d77f2bb3914131a0" gracePeriod=30 Nov 02 20:25:38.278249 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:38.278211 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:38.684738 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:38.684709 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw"] Nov 02 20:25:38.685238 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:38.684954 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerName="istio-proxy" containerID="cri-o://1168528efc2ccb9dfb3129b4326a2bbaf9cde2b10857cb6c7bfac11a38e6a1c1" gracePeriod=30 Nov 02 20:25:38.732333 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:38.732298 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w"] Nov 02 20:25:38.732521 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:38.732479 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" containerID="cri-o://1688b9bd2353fbb1651199c4c9c6b602a622fc84e441bbbfa68d30a150bfee36" gracePeriod=30 Nov 02 20:25:42.513225 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:42.513188 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:43.005522 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.005175 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:25:43.005522 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.005253 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:25:43.037185 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.037150 5561 generic.go:356] "Generic (PLEG): container finished" podID="cdb9d013-92c8-426b-9127-eaac2797978f" containerID="709a4646e23453c1ea631b0dfbfbbe5ac188355a47adaea53a319cb0b1bc07f0" exitCode=0 Nov 02 20:25:43.037365 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.037290 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" event={"ID":"cdb9d013-92c8-426b-9127-eaac2797978f","Type":"ContainerDied","Data":"709a4646e23453c1ea631b0dfbfbbe5ac188355a47adaea53a319cb0b1bc07f0"} Nov 02 20:25:43.039414 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.039272 5561 generic.go:356] "Generic (PLEG): container finished" podID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerID="f5df2de9ae8ae1df2d32c983e0670cd2e5fdd26b3982fb77d77f2bb3914131a0" exitCode=0 Nov 02 20:25:43.039414 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.039370 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" event={"ID":"88985b8c-444e-4b57-b179-2e114dfa7d48","Type":"ContainerDied","Data":"f5df2de9ae8ae1df2d32c983e0670cd2e5fdd26b3982fb77d77f2bb3914131a0"} Nov 02 20:25:43.217622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.217582 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:25:43.243644 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.243620 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:25:43.280376 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.280348 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-data\") pod \"88985b8c-444e-4b57-b179-2e114dfa7d48\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " Nov 02 20:25:43.280565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.280395 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-credential-socket\") pod \"88985b8c-444e-4b57-b179-2e114dfa7d48\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " Nov 02 20:25:43.280565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.280444 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb9d013-92c8-426b-9127-eaac2797978f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb9d013-92c8-426b-9127-eaac2797978f-istiod-ca-cert\") pod \"cdb9d013-92c8-426b-9127-eaac2797978f\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " Nov 02 20:25:43.280565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.280486 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-kube-api-access-4pqdd\" (UniqueName: \"kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-kube-api-access-4pqdd\") pod \"cdb9d013-92c8-426b-9127-eaac2797978f\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " Nov 02 20:25:43.280565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.280521 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-istio-token\" (UniqueName: \"kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-istio-token\") pod \"cdb9d013-92c8-426b-9127-eaac2797978f\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " Nov 02 20:25:43.280565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.280539 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-envoy\") pod \"88985b8c-444e-4b57-b179-2e114dfa7d48\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " Nov 02 20:25:43.280565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.280564 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-data\") pod \"cdb9d013-92c8-426b-9127-eaac2797978f\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " Nov 02 20:25:43.280864 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.280630 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-data" pod "88985b8c-444e-4b57-b179-2e114dfa7d48" (UID: "88985b8c-444e-4b57-b179-2e114dfa7d48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:43.280864 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.280731 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-credential-socket" pod "88985b8c-444e-4b57-b179-2e114dfa7d48" (UID: "88985b8c-444e-4b57-b179-2e114dfa7d48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:43.280864 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.280805 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb9d013-92c8-426b-9127-eaac2797978f-istiod-ca-cert" pod "cdb9d013-92c8-426b-9127-eaac2797978f" (UID: "cdb9d013-92c8-426b-9127-eaac2797978f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:43.281104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.280970 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-data" pod "cdb9d013-92c8-426b-9127-eaac2797978f" (UID: "cdb9d013-92c8-426b-9127-eaac2797978f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:43.281104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.281064 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdb9d013-92c8-426b-9127-eaac2797978f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb9d013-92c8-426b-9127-eaac2797978f-istio-podinfo\") pod \"cdb9d013-92c8-426b-9127-eaac2797978f\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " Nov 02 20:25:43.281196 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.281152 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-kube-api-access-bh72m\" (UniqueName: \"kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-kube-api-access-bh72m\") pod \"88985b8c-444e-4b57-b179-2e114dfa7d48\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " Nov 02 20:25:43.281196 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.281176 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-credential-socket\") pod \"cdb9d013-92c8-426b-9127-eaac2797978f\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " Nov 02 20:25:43.281296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.281209 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-envoy\") pod \"cdb9d013-92c8-426b-9127-eaac2797978f\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " Nov 02 20:25:43.281296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.281238 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-token\" (UniqueName: \"kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-token\") pod \"88985b8c-444e-4b57-b179-2e114dfa7d48\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " Nov 02 20:25:43.281296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.281268 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-certs\") pod \"cdb9d013-92c8-426b-9127-eaac2797978f\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " Nov 02 20:25:43.281443 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.281300 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-socket\") pod \"cdb9d013-92c8-426b-9127-eaac2797978f\" (UID: \"cdb9d013-92c8-426b-9127-eaac2797978f\") " Nov 02 20:25:43.281443 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.281361 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-socket\") pod \"88985b8c-444e-4b57-b179-2e114dfa7d48\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " Nov 02 20:25:43.281544 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.281440 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88985b8c-444e-4b57-b179-2e114dfa7d48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88985b8c-444e-4b57-b179-2e114dfa7d48-istiod-ca-cert\") pod \"88985b8c-444e-4b57-b179-2e114dfa7d48\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " Nov 02 20:25:43.281544 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.281491 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-podinfo\") pod \"88985b8c-444e-4b57-b179-2e114dfa7d48\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " Nov 02 20:25:43.281544 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.281534 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-certs\") pod \"88985b8c-444e-4b57-b179-2e114dfa7d48\" (UID: \"88985b8c-444e-4b57-b179-2e114dfa7d48\") " Nov 02 20:25:43.281719 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.281694 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-socket" pod "cdb9d013-92c8-426b-9127-eaac2797978f" (UID: "cdb9d013-92c8-426b-9127-eaac2797978f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:43.282133 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.282103 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-credential-socket" pod "cdb9d013-92c8-426b-9127-eaac2797978f" (UID: "cdb9d013-92c8-426b-9127-eaac2797978f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:43.282322 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.282297 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-certs" pod "cdb9d013-92c8-426b-9127-eaac2797978f" (UID: "cdb9d013-92c8-426b-9127-eaac2797978f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:43.282380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.282333 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.282380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.282356 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.282380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.282371 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb9d013-92c8-426b-9127-eaac2797978f-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.282528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.282385 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.282528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.282403 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.282528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.282417 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.282528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.282460 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-socket" pod "88985b8c-444e-4b57-b179-2e114dfa7d48" (UID: "88985b8c-444e-4b57-b179-2e114dfa7d48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:43.282725 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.282556 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88985b8c-444e-4b57-b179-2e114dfa7d48-istiod-ca-cert" pod "88985b8c-444e-4b57-b179-2e114dfa7d48" (UID: "88985b8c-444e-4b57-b179-2e114dfa7d48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:43.282777 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.282747 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-certs" pod "88985b8c-444e-4b57-b179-2e114dfa7d48" (UID: "88985b8c-444e-4b57-b179-2e114dfa7d48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:43.283605 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.283578 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdb9d013-92c8-426b-9127-eaac2797978f-istio-podinfo" pod "cdb9d013-92c8-426b-9127-eaac2797978f" (UID: "cdb9d013-92c8-426b-9127-eaac2797978f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:25:43.284089 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.284067 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-envoy" pod "88985b8c-444e-4b57-b179-2e114dfa7d48" (UID: "88985b8c-444e-4b57-b179-2e114dfa7d48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:43.284167 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.284095 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-istio-token" pod "cdb9d013-92c8-426b-9127-eaac2797978f" (UID: "cdb9d013-92c8-426b-9127-eaac2797978f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:43.284219 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.284163 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-envoy" pod "cdb9d013-92c8-426b-9127-eaac2797978f" (UID: "cdb9d013-92c8-426b-9127-eaac2797978f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:43.284663 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.284630 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-token" pod "88985b8c-444e-4b57-b179-2e114dfa7d48" (UID: "88985b8c-444e-4b57-b179-2e114dfa7d48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:43.284781 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.284731 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-kube-api-access-bh72m" pod "88985b8c-444e-4b57-b179-2e114dfa7d48" (UID: "88985b8c-444e-4b57-b179-2e114dfa7d48"). InnerVolumeSpecName "kube-api-access-bh72m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:43.285157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.285130 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-kube-api-access-4pqdd" pod "cdb9d013-92c8-426b-9127-eaac2797978f" (UID: "cdb9d013-92c8-426b-9127-eaac2797978f"). InnerVolumeSpecName "kube-api-access-4pqdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:43.285446 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.285430 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-podinfo" pod "88985b8c-444e-4b57-b179-2e114dfa7d48" (UID: "88985b8c-444e-4b57-b179-2e114dfa7d48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:25:43.290211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.290191 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:43.411005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.405336 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pqdd\" (UniqueName: \"kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-kube-api-access-4pqdd\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.411005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.405375 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb9d013-92c8-426b-9127-eaac2797978f-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.411005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.405391 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.411005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.405405 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb9d013-92c8-426b-9127-eaac2797978f-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.411005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.405419 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh72m\" (UniqueName: \"kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-kube-api-access-bh72m\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.411005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.405434 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.411005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.405448 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.411005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.405461 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb9d013-92c8-426b-9127-eaac2797978f-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.411005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.405474 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.411005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.405487 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88985b8c-444e-4b57-b179-2e114dfa7d48-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.411005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.405502 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88985b8c-444e-4b57-b179-2e114dfa7d48-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.411005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.405516 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88985b8c-444e-4b57-b179-2e114dfa7d48-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:43.537895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.533654 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:43.670252 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.670171 5561 scope.go:122] "RemoveContainer" containerID="709a4646e23453c1ea631b0dfbfbbe5ac188355a47adaea53a319cb0b1bc07f0" Nov 02 20:25:43.684088 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.684062 5561 scope.go:122] "RemoveContainer" containerID="f5df2de9ae8ae1df2d32c983e0670cd2e5fdd26b3982fb77d77f2bb3914131a0" Nov 02 20:25:43.716363 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:43.716323 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:44.065900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.059011 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" event={"ID":"cdb9d013-92c8-426b-9127-eaac2797978f","Type":"ContainerDied","Data":"fe8d2c609c5fceb094c88bab6600af09cbabb9068bd45fbf4816a0de93f88bcd"} Nov 02 20:25:44.069062 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.068905 5561 generic.go:356] "Generic (PLEG): container finished" podID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerID="1168528efc2ccb9dfb3129b4326a2bbaf9cde2b10857cb6c7bfac11a38e6a1c1" exitCode=0 Nov 02 20:25:44.069220 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.069142 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" event={"ID":"3abaaa8a-f947-4d2a-b73a-424c4ff05f84","Type":"ContainerDied","Data":"1168528efc2ccb9dfb3129b4326a2bbaf9cde2b10857cb6c7bfac11a38e6a1c1"} Nov 02 20:25:44.069286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.069216 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" event={"ID":"88985b8c-444e-4b57-b179-2e114dfa7d48","Type":"ContainerDied","Data":"1a309b71bfd4be8488ef05db3105f9850d726692bfb6226719578618374db563"} Nov 02 20:25:44.078178 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.072484 5561 generic.go:356] "Generic (PLEG): container finished" podID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerID="1688b9bd2353fbb1651199c4c9c6b602a622fc84e441bbbfa68d30a150bfee36" exitCode=0 Nov 02 20:25:44.078178 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.072632 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p" Nov 02 20:25:44.078178 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.073304 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" event={"ID":"da8e9e7a-7797-4186-916f-15c6e80b7098","Type":"ContainerDied","Data":"1688b9bd2353fbb1651199c4c9c6b602a622fc84e441bbbfa68d30a150bfee36"} Nov 02 20:25:44.078178 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.073389 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6" Nov 02 20:25:44.160145 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.158128 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:44.163067 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.162983 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:25:44.168247 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.168222 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-envoy\") pod \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " Nov 02 20:25:44.168379 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.168259 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-data\") pod \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " Nov 02 20:25:44.168379 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.168314 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istiod-ca-cert\") pod \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " Nov 02 20:25:44.168379 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.168348 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-token\" (UniqueName: \"kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-token\") pod \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " Nov 02 20:25:44.168554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.168460 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-credential-socket\") pod \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " Nov 02 20:25:44.168554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.168486 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-socket\") pod \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " Nov 02 20:25:44.168554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.168545 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-kube-api-access-w5c7b\" (UniqueName: \"kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-kube-api-access-w5c7b\") pod \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " Nov 02 20:25:44.168716 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.168599 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-podinfo\") pod \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " Nov 02 20:25:44.168716 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.168642 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-certs\") pod \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\" (UID: \"3abaaa8a-f947-4d2a-b73a-424c4ff05f84\") " Nov 02 20:25:44.169198 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.169133 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-socket" pod "3abaaa8a-f947-4d2a-b73a-424c4ff05f84" (UID: "3abaaa8a-f947-4d2a-b73a-424c4ff05f84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.169198 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.169147 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-credential-socket" pod "3abaaa8a-f947-4d2a-b73a-424c4ff05f84" (UID: "3abaaa8a-f947-4d2a-b73a-424c4ff05f84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.169468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.169436 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-data" pod "3abaaa8a-f947-4d2a-b73a-424c4ff05f84" (UID: "3abaaa8a-f947-4d2a-b73a-424c4ff05f84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.170042 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.170014 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-certs" pod "3abaaa8a-f947-4d2a-b73a-424c4ff05f84" (UID: "3abaaa8a-f947-4d2a-b73a-424c4ff05f84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.170253 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.170232 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istiod-ca-cert" pod "3abaaa8a-f947-4d2a-b73a-424c4ff05f84" (UID: "3abaaa8a-f947-4d2a-b73a-424c4ff05f84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:44.170923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.170803 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.170923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.170827 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.170923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.170842 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.170923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.170857 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.170923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.170903 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.172024 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.172001 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-podinfo" pod "3abaaa8a-f947-4d2a-b73a-424c4ff05f84" (UID: "3abaaa8a-f947-4d2a-b73a-424c4ff05f84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:25:44.172347 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.172329 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-envoy" pod "3abaaa8a-f947-4d2a-b73a-424c4ff05f84" (UID: "3abaaa8a-f947-4d2a-b73a-424c4ff05f84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.173968 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.173945 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-token" pod "3abaaa8a-f947-4d2a-b73a-424c4ff05f84" (UID: "3abaaa8a-f947-4d2a-b73a-424c4ff05f84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:44.175013 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.174991 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-kube-api-access-w5c7b" pod "3abaaa8a-f947-4d2a-b73a-424c4ff05f84" (UID: "3abaaa8a-f947-4d2a-b73a-424c4ff05f84"). InnerVolumeSpecName "kube-api-access-w5c7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:44.239847 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.239820 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:25:44.271592 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.271492 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-socket\") pod \"da8e9e7a-7797-4186-916f-15c6e80b7098\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " Nov 02 20:25:44.271766 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.271657 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-token\" (UniqueName: \"kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-token\") pod \"da8e9e7a-7797-4186-916f-15c6e80b7098\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " Nov 02 20:25:44.271847 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.271767 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-kube-api-access-wnkkp\" (UniqueName: \"kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-kube-api-access-wnkkp\") pod \"da8e9e7a-7797-4186-916f-15c6e80b7098\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " Nov 02 20:25:44.271847 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.271820 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-socket" pod "da8e9e7a-7797-4186-916f-15c6e80b7098" (UID: "da8e9e7a-7797-4186-916f-15c6e80b7098"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.272119 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.272093 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-credential-socket" pod "da8e9e7a-7797-4186-916f-15c6e80b7098" (UID: "da8e9e7a-7797-4186-916f-15c6e80b7098"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.272400 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.272379 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-credential-socket\") pod \"da8e9e7a-7797-4186-916f-15c6e80b7098\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " Nov 02 20:25:44.272704 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.272680 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-certs\") pod \"da8e9e7a-7797-4186-916f-15c6e80b7098\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " Nov 02 20:25:44.272810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.272790 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-data\") pod \"da8e9e7a-7797-4186-916f-15c6e80b7098\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " Nov 02 20:25:44.272951 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.272862 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-podinfo\") pod \"da8e9e7a-7797-4186-916f-15c6e80b7098\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " Nov 02 20:25:44.272951 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.272918 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-envoy\") pod \"da8e9e7a-7797-4186-916f-15c6e80b7098\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " Nov 02 20:25:44.272951 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.272948 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8e9e7a-7797-4186-916f-15c6e80b7098-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8e9e7a-7797-4186-916f-15c6e80b7098-istiod-ca-cert\") pod \"da8e9e7a-7797-4186-916f-15c6e80b7098\" (UID: \"da8e9e7a-7797-4186-916f-15c6e80b7098\") " Nov 02 20:25:44.273125 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.272982 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-certs" pod "da8e9e7a-7797-4186-916f-15c6e80b7098" (UID: "da8e9e7a-7797-4186-916f-15c6e80b7098"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.273630 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.273595 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.273630 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.273620 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.273780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.273635 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.273780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.273650 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.273780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.273666 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5c7b\" (UniqueName: \"kubernetes.io/projected/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-kube-api-access-w5c7b\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.273780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.273681 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3abaaa8a-f947-4d2a-b73a-424c4ff05f84-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.273780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.273683 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8e9e7a-7797-4186-916f-15c6e80b7098-istiod-ca-cert" pod "da8e9e7a-7797-4186-916f-15c6e80b7098" (UID: "da8e9e7a-7797-4186-916f-15c6e80b7098"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:44.273780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.273694 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.274202 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.274178 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-data" pod "da8e9e7a-7797-4186-916f-15c6e80b7098" (UID: "da8e9e7a-7797-4186-916f-15c6e80b7098"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.274425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.274377 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-token" pod "da8e9e7a-7797-4186-916f-15c6e80b7098" (UID: "da8e9e7a-7797-4186-916f-15c6e80b7098"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:44.274608 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.274524 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-kube-api-access-wnkkp" pod "da8e9e7a-7797-4186-916f-15c6e80b7098" (UID: "da8e9e7a-7797-4186-916f-15c6e80b7098"). InnerVolumeSpecName "kube-api-access-wnkkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:44.275494 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.275468 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-podinfo" pod "da8e9e7a-7797-4186-916f-15c6e80b7098" (UID: "da8e9e7a-7797-4186-916f-15c6e80b7098"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:25:44.276652 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.276625 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-envoy" pod "da8e9e7a-7797-4186-916f-15c6e80b7098" (UID: "da8e9e7a-7797-4186-916f-15c6e80b7098"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.375108 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.375074 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnkkp\" (UniqueName: \"kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-kube-api-access-wnkkp\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.375108 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.375104 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.375318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.375125 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.375318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.375134 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.375318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.375143 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8e9e7a-7797-4186-916f-15c6e80b7098-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.375318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.375152 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8e9e7a-7797-4186-916f-15c6e80b7098-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.430467 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.430430 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:44.528596 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.528507 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p"] Nov 02 20:25:44.532976 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.532949 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-52x9p"] Nov 02 20:25:44.622477 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.622441 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:44.745103 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.743963 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6"] Nov 02 20:25:44.750478 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.750446 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-6dkq6"] Nov 02 20:25:44.824981 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:44.824904 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:45.080891 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.080806 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" Nov 02 20:25:45.080891 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.080832 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w" event={"ID":"da8e9e7a-7797-4186-916f-15c6e80b7098","Type":"ContainerDied","Data":"d537672b74f114e65616d21ae75d24b0213782d551d6fd9668a7d55cd1114e95"} Nov 02 20:25:45.081092 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.080963 5561 scope.go:122] "RemoveContainer" containerID="1688b9bd2353fbb1651199c4c9c6b602a622fc84e441bbbfa68d30a150bfee36" Nov 02 20:25:45.082427 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.082406 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" event={"ID":"3abaaa8a-f947-4d2a-b73a-424c4ff05f84","Type":"ContainerDied","Data":"1c2b00e8e165816fc90632ebc3992462b1c36f8cd7c0efe83ba659ced3221076"} Nov 02 20:25:45.082523 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.082421 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw" Nov 02 20:25:45.098597 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.098574 5561 scope.go:122] "RemoveContainer" containerID="1168528efc2ccb9dfb3129b4326a2bbaf9cde2b10857cb6c7bfac11a38e6a1c1" Nov 02 20:25:45.246485 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.246226 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w"] Nov 02 20:25:45.287645 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.287612 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-t2j5w"] Nov 02 20:25:45.315617 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.315588 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" path="/var/lib/kubelet/pods/88985b8c-444e-4b57-b179-2e114dfa7d48/volumes" Nov 02 20:25:45.316134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.316119 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" path="/var/lib/kubelet/pods/cdb9d013-92c8-426b-9127-eaac2797978f/volumes" Nov 02 20:25:45.316546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.316534 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" path="/var/lib/kubelet/pods/da8e9e7a-7797-4186-916f-15c6e80b7098/volumes" Nov 02 20:25:45.355324 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.355197 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw"] Nov 02 20:25:45.359681 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.359652 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-p77lw"] Nov 02 20:25:45.973543 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:45.973507 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:46.380005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:46.379870 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:47.314442 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:47.314413 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" path="/var/lib/kubelet/pods/3abaaa8a-f947-4d2a-b73a-424c4ff05f84/volumes" Nov 02 20:25:48.111407 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:48.111368 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:48.292862 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:48.292825 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:48.588981 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:48.588940 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:49.513152 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:49.513001 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:49.753730 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:49.753694 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:49.933440 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:49.933411 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:50.868122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:50.868077 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:51.056555 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:51.056517 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:52.204061 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:52.204014 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:52.454456 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:52.454369 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z"] Nov 02 20:25:52.454661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:52.454619 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" podUID="bd7b76c6-3cb5-48f1-8347-2fe977212038" containerName="istio-proxy" containerID="cri-o://f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35" gracePeriod=30 Nov 02 20:25:52.662277 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:52.662243 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:53.439221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:53.439186 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:53.999141 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:53.999109 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:54.179813 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.179779 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:54.387273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.387185 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:54.565349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.565306 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:54.735299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.735259 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:25:54.735507 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.735332 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" podUID="bd7b76c6-3cb5-48f1-8347-2fe977212038" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:25:54.778104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778069 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq"] Nov 02 20:25:54.778618 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778605 5561 cpu_manager.go:414] "containerMap: removing container" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerName="istio-proxy" Nov 02 20:25:54.778661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778620 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerName="istio-proxy" Nov 02 20:25:54.778661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778638 5561 cpu_manager.go:414] "containerMap: removing container" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" Nov 02 20:25:54.778661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778642 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" Nov 02 20:25:54.778661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778654 5561 cpu_manager.go:414] "containerMap: removing container" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" Nov 02 20:25:54.778661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778659 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" Nov 02 20:25:54.778816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778680 5561 cpu_manager.go:414] "containerMap: removing container" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" Nov 02 20:25:54.778816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778686 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" Nov 02 20:25:54.778816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778780 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb9d013-92c8-426b-9127-eaac2797978f" containerName="istio-proxy" Nov 02 20:25:54.778816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778789 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abaaa8a-f947-4d2a-b73a-424c4ff05f84" containerName="istio-proxy" Nov 02 20:25:54.778816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778798 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8e9e7a-7797-4186-916f-15c6e80b7098" containerName="istio-proxy" Nov 02 20:25:54.778816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.778805 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="88985b8c-444e-4b57-b179-2e114dfa7d48" containerName="istio-proxy" Nov 02 20:25:54.783473 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.783454 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" Nov 02 20:25:54.835942 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.835907 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq"] Nov 02 20:25:54.839212 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.839186 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-abc937\"" Nov 02 20:25:54.839212 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.839186 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-abc937-dockercfg-4mr2h\"" Nov 02 20:25:54.937355 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.937318 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d57d\" (UniqueName: \"kubernetes.io/projected/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-kube-api-access-2d57d\") pod \"payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq\" (UID: \"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" Nov 02 20:25:54.937706 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:54.937679 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq\" (UID: \"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" Nov 02 20:25:55.039643 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:55.039559 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d57d\" (UniqueName: \"kubernetes.io/projected/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-kube-api-access-2d57d\") pod \"payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq\" (UID: \"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" Nov 02 20:25:55.039798 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:55.039642 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq\" (UID: \"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" Nov 02 20:25:55.040315 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:55.040295 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq\" (UID: \"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" Nov 02 20:25:55.094558 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:55.094523 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d57d\" (UniqueName: \"kubernetes.io/projected/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-kube-api-access-2d57d\") pod \"payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq\" (UID: \"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" Nov 02 20:25:55.097161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:55.097135 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" Nov 02 20:25:55.477677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:55.477636 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq"] Nov 02 20:25:55.484226 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:25:55.484197 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1e0f45_5cb6_4b98_b7db_4df3b3abcd0a.slice/crio-c9b4479ad3a71b07bfeded66ed9a286df88d67f93ae651292156f9e71e718153 WatchSource:0}: Error finding container c9b4479ad3a71b07bfeded66ed9a286df88d67f93ae651292156f9e71e718153: Status 404 returned error can't find the container with id c9b4479ad3a71b07bfeded66ed9a286df88d67f93ae651292156f9e71e718153 Nov 02 20:25:55.486588 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:55.486571 5561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:25:56.168101 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:56.168067 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" event={"ID":"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a","Type":"ContainerStarted","Data":"c9b4479ad3a71b07bfeded66ed9a286df88d67f93ae651292156f9e71e718153"} Nov 02 20:25:57.726545 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.726487 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:25:57.880617 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.880583 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7b76c6-3cb5-48f1-8347-2fe977212038-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd7b76c6-3cb5-48f1-8347-2fe977212038-istiod-ca-cert\") pod \"bd7b76c6-3cb5-48f1-8347-2fe977212038\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " Nov 02 20:25:57.880809 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.880688 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-podinfo\") pod \"bd7b76c6-3cb5-48f1-8347-2fe977212038\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " Nov 02 20:25:57.880809 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.880787 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-data\") pod \"bd7b76c6-3cb5-48f1-8347-2fe977212038\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " Nov 02 20:25:57.880959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.880905 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-token\" (UniqueName: \"kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-token\") pod \"bd7b76c6-3cb5-48f1-8347-2fe977212038\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " Nov 02 20:25:57.880959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.880938 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-credential-socket\") pod \"bd7b76c6-3cb5-48f1-8347-2fe977212038\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " Nov 02 20:25:57.881074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.880972 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7b76c6-3cb5-48f1-8347-2fe977212038-istiod-ca-cert" pod "bd7b76c6-3cb5-48f1-8347-2fe977212038" (UID: "bd7b76c6-3cb5-48f1-8347-2fe977212038"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:57.881074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.880992 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-kube-api-access-29fq5\" (UniqueName: \"kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-kube-api-access-29fq5\") pod \"bd7b76c6-3cb5-48f1-8347-2fe977212038\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " Nov 02 20:25:57.881074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.881053 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-envoy\") pod \"bd7b76c6-3cb5-48f1-8347-2fe977212038\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " Nov 02 20:25:57.881217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.881082 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-socket\") pod \"bd7b76c6-3cb5-48f1-8347-2fe977212038\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " Nov 02 20:25:57.881217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.881115 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-data" pod "bd7b76c6-3cb5-48f1-8347-2fe977212038" (UID: "bd7b76c6-3cb5-48f1-8347-2fe977212038"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:57.881217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.881183 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-certs\") pod \"bd7b76c6-3cb5-48f1-8347-2fe977212038\" (UID: \"bd7b76c6-3cb5-48f1-8347-2fe977212038\") " Nov 02 20:25:57.881366 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.881251 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-credential-socket" pod "bd7b76c6-3cb5-48f1-8347-2fe977212038" (UID: "bd7b76c6-3cb5-48f1-8347-2fe977212038"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:57.881482 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.881452 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-socket" pod "bd7b76c6-3cb5-48f1-8347-2fe977212038" (UID: "bd7b76c6-3cb5-48f1-8347-2fe977212038"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:57.881613 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.881548 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-certs" pod "bd7b76c6-3cb5-48f1-8347-2fe977212038" (UID: "bd7b76c6-3cb5-48f1-8347-2fe977212038"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:57.881861 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.881839 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:57.881972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.881866 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd7b76c6-3cb5-48f1-8347-2fe977212038-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:57.881972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.881897 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:57.881972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.881912 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:57.881972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.881928 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:57.883475 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.883451 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-podinfo" pod "bd7b76c6-3cb5-48f1-8347-2fe977212038" (UID: "bd7b76c6-3cb5-48f1-8347-2fe977212038"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:25:57.883559 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.883470 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-token" pod "bd7b76c6-3cb5-48f1-8347-2fe977212038" (UID: "bd7b76c6-3cb5-48f1-8347-2fe977212038"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:57.883559 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.883513 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-kube-api-access-29fq5" pod "bd7b76c6-3cb5-48f1-8347-2fe977212038" (UID: "bd7b76c6-3cb5-48f1-8347-2fe977212038"). InnerVolumeSpecName "kube-api-access-29fq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:57.885154 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.885111 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-envoy" pod "bd7b76c6-3cb5-48f1-8347-2fe977212038" (UID: "bd7b76c6-3cb5-48f1-8347-2fe977212038"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:57.983285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.983200 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:57.983285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.983240 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29fq5\" (UniqueName: \"kubernetes.io/projected/bd7b76c6-3cb5-48f1-8347-2fe977212038-kube-api-access-29fq5\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:57.983285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.983256 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:57.983285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:57.983269 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd7b76c6-3cb5-48f1-8347-2fe977212038-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:58.149473 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:58.149436 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:58.181980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:58.181945 5561 generic.go:356] "Generic (PLEG): container finished" podID="bd7b76c6-3cb5-48f1-8347-2fe977212038" containerID="f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35" exitCode=0 Nov 02 20:25:58.182157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:58.181998 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" Nov 02 20:25:58.182157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:58.182033 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" event={"ID":"bd7b76c6-3cb5-48f1-8347-2fe977212038","Type":"ContainerDied","Data":"f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35"} Nov 02 20:25:58.182157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:58.182087 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z" event={"ID":"bd7b76c6-3cb5-48f1-8347-2fe977212038","Type":"ContainerDied","Data":"381b42f78e866c9636b3f621c00081c8762b2b6eeae71a7cb9185ce133daef41"} Nov 02 20:25:58.182157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:58.182109 5561 scope.go:122] "RemoveContainer" containerID="f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35" Nov 02 20:25:58.193192 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:58.193173 5561 scope.go:122] "RemoveContainer" containerID="f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35" Nov 02 20:25:58.193484 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:25:58.193466 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35\": container with ID starting with f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35 not found: ID does not exist" containerID="f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35" Nov 02 20:25:58.193526 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:58.193495 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35"} err="failed to get container status \"f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35\": rpc error: code = NotFound desc = could not find container \"f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35\": container with ID starting with f236a5f34622295ce51026ff28bb4497d77e22fde9fa85fa52cbfec21a10bd35 not found: ID does not exist" Nov 02 20:25:58.325543 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:58.325456 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:58.338687 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:58.338653 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z"] Nov 02 20:25:58.341266 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:58.341242 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-xjp7z"] Nov 02 20:25:59.095682 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.095616 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:59.320462 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.320420 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7b76c6-3cb5-48f1-8347-2fe977212038" path="/var/lib/kubelet/pods/bd7b76c6-3cb5-48f1-8347-2fe977212038/volumes" Nov 02 20:25:59.325240 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.325209 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:59.468391 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.468352 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4"] Nov 02 20:25:59.469027 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.469004 5561 cpu_manager.go:414] "containerMap: removing container" podUID="bd7b76c6-3cb5-48f1-8347-2fe977212038" containerName="istio-proxy" Nov 02 20:25:59.469421 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.469028 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7b76c6-3cb5-48f1-8347-2fe977212038" containerName="istio-proxy" Nov 02 20:25:59.469421 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.469180 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7b76c6-3cb5-48f1-8347-2fe977212038" containerName="istio-proxy" Nov 02 20:25:59.474737 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.474702 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" Nov 02 20:25:59.507122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.507083 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:59.525238 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.525203 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4"] Nov 02 20:25:59.529023 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.528854 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-602be312\"" Nov 02 20:25:59.582229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.582151 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-602be312-dockercfg-s7gt8\"" Nov 02 20:25:59.609360 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.609310 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wt2r\" (UniqueName: \"kubernetes.io/projected/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-kube-api-access-7wt2r\") pod \"payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4\" (UID: \"b6051c83-da4e-482c-8833-ca0dd2bc9bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" Nov 02 20:25:59.610308 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.610265 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4\" (UID: \"b6051c83-da4e-482c-8833-ca0dd2bc9bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" Nov 02 20:25:59.630932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.619666 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl"] Nov 02 20:25:59.655948 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.655320 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" Nov 02 20:25:59.677897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.676627 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl"] Nov 02 20:25:59.712636 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.712079 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wt2r\" (UniqueName: \"kubernetes.io/projected/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-kube-api-access-7wt2r\") pod \"payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4\" (UID: \"b6051c83-da4e-482c-8833-ca0dd2bc9bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" Nov 02 20:25:59.712636 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.712197 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4\" (UID: \"b6051c83-da4e-482c-8833-ca0dd2bc9bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" Nov 02 20:25:59.713100 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.713080 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4\" (UID: \"b6051c83-da4e-482c-8833-ca0dd2bc9bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" Nov 02 20:25:59.773330 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.773254 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wt2r\" (UniqueName: \"kubernetes.io/projected/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-kube-api-access-7wt2r\") pod \"payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4\" (UID: \"b6051c83-da4e-482c-8833-ca0dd2bc9bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" Nov 02 20:25:59.807212 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.804034 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" Nov 02 20:25:59.822230 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.821842 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck96g\" (UniqueName: \"kubernetes.io/projected/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-kube-api-access-ck96g\") pod \"payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl\" (UID: \"6e8276a3-e24b-4a21-8522-c5bac20d5a3b\") " pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" Nov 02 20:25:59.822230 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.822091 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-plugins-config-volume\") pod \"payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl\" (UID: \"6e8276a3-e24b-4a21-8522-c5bac20d5a3b\") " pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" Nov 02 20:25:59.929915 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.923957 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-plugins-config-volume\") pod \"payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl\" (UID: \"6e8276a3-e24b-4a21-8522-c5bac20d5a3b\") " pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" Nov 02 20:25:59.929915 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.924911 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-plugins-config-volume\") pod \"payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl\" (UID: \"6e8276a3-e24b-4a21-8522-c5bac20d5a3b\") " pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" Nov 02 20:25:59.929915 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.925103 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck96g\" (UniqueName: \"kubernetes.io/projected/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-kube-api-access-ck96g\") pod \"payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl\" (UID: \"6e8276a3-e24b-4a21-8522-c5bac20d5a3b\") " pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" Nov 02 20:25:59.981273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:25:59.981243 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck96g\" (UniqueName: \"kubernetes.io/projected/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-kube-api-access-ck96g\") pod \"payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl\" (UID: \"6e8276a3-e24b-4a21-8522-c5bac20d5a3b\") " pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" Nov 02 20:26:00.012079 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:00.012048 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" Nov 02 20:26:00.225833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:00.225794 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4"] Nov 02 20:26:00.236037 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:26:00.235978 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6051c83_da4e_482c_8833_ca0dd2bc9bcf.slice/crio-12638880d890891e40a77e521dd0f0ec866b70ff7992946274128961ce5f554b WatchSource:0}: Error finding container 12638880d890891e40a77e521dd0f0ec866b70ff7992946274128961ce5f554b: Status 404 returned error can't find the container with id 12638880d890891e40a77e521dd0f0ec866b70ff7992946274128961ce5f554b Nov 02 20:26:00.398674 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:00.398598 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl"] Nov 02 20:26:00.405065 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:26:00.405031 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8276a3_e24b_4a21_8522_c5bac20d5a3b.slice/crio-c5b3da5686966a07a8aa3f01ba61ed8dd9421765301e78796c230c13257a7b99 WatchSource:0}: Error finding container c5b3da5686966a07a8aa3f01ba61ed8dd9421765301e78796c230c13257a7b99: Status 404 returned error can't find the container with id c5b3da5686966a07a8aa3f01ba61ed8dd9421765301e78796c230c13257a7b99 Nov 02 20:26:01.246084 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:01.246046 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" event={"ID":"6e8276a3-e24b-4a21-8522-c5bac20d5a3b","Type":"ContainerStarted","Data":"c5b3da5686966a07a8aa3f01ba61ed8dd9421765301e78796c230c13257a7b99"} Nov 02 20:26:01.247633 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:01.247606 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" event={"ID":"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a","Type":"ContainerStarted","Data":"2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2"} Nov 02 20:26:01.247777 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:01.247679 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" Nov 02 20:26:01.249339 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:01.249315 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" event={"ID":"b6051c83-da4e-482c-8833-ca0dd2bc9bcf","Type":"ContainerStarted","Data":"72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47"} Nov 02 20:26:01.249465 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:01.249355 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" event={"ID":"b6051c83-da4e-482c-8833-ca0dd2bc9bcf","Type":"ContainerStarted","Data":"12638880d890891e40a77e521dd0f0ec866b70ff7992946274128961ce5f554b"} Nov 02 20:26:01.249465 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:01.249419 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" Nov 02 20:26:01.356685 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:01.356633 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" podStartSLOduration=2.119460176 podStartE2EDuration="7.356619684s" totalImagesPullingTime="5.237159508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:01.355954149 +0000 UTC m=+1998.163738365" watchObservedRunningTime="2026-11-02 20:26:01.356619684 +0000 UTC m=+1998.164403900" Nov 02 20:26:01.477932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:01.465814 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" podStartSLOduration=1.630688068 podStartE2EDuration="2.465793324s" totalImagesPullingTime="835.105256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:01.462570414 +0000 UTC m=+1998.270354651" watchObservedRunningTime="2026-11-02 20:26:01.465793324 +0000 UTC m=+1998.273577542" Nov 02 20:26:02.257699 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:02.257656 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" event={"ID":"6e8276a3-e24b-4a21-8522-c5bac20d5a3b","Type":"ContainerStarted","Data":"3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a"} Nov 02 20:26:02.367040 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:02.366988 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" podStartSLOduration=2.5088215270000003 podStartE2EDuration="3.366975824s" totalImagesPullingTime="858.154297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:02.364758709 +0000 UTC m=+1999.172542945" watchObservedRunningTime="2026-11-02 20:26:02.366975824 +0000 UTC m=+1999.174760041" Nov 02 20:26:03.263468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:03.263438 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" Nov 02 20:26:03.887192 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:26:03.887149 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:26:04.543274 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:04.542860 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:09.373899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:09.370744 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:12.260969 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:12.260938 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" Nov 02 20:26:12.261401 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:12.261156 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" Nov 02 20:26:13.895418 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:13.888036 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:14.272315 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:14.272287 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" Nov 02 20:26:15.088626 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:15.088589 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:15.257357 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:15.257328 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:18.240418 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:18.240379 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:18.422997 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:18.422961 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:18.594319 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:18.594243 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:23.034986 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:23.034950 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:27.413627 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:27.413520 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:27.588302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:27.588259 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:27.793838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:27.793760 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:32.007734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:32.007697 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:33.779596 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:26:33.779561 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:26:36.583969 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:36.583933 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:41.036789 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:41.036729 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:42.778898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:42.777397 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:45.366410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.366375 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq"] Nov 02 20:26:45.371833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.371813 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.381357 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.381322 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-maas-api-tls\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.381479 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.381407 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmxjf\" (UniqueName: \"kubernetes.io/projected/43a5c044-6e8d-4c38-8665-809d63fdf010-kube-api-access-pmxjf\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.381531 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.381509 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-metrics-certs\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.381568 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.381555 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a5c044-6e8d-4c38-8665-809d63fdf010-openshift-service-ca\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.424312 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.424275 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq"] Nov 02 20:26:45.467346 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.467312 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-eb060a-serving-cert\"" Nov 02 20:26:45.483083 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.483049 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmxjf\" (UniqueName: \"kubernetes.io/projected/43a5c044-6e8d-4c38-8665-809d63fdf010-kube-api-access-pmxjf\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.483255 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.483106 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-metrics-certs\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.483255 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.483135 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a5c044-6e8d-4c38-8665-809d63fdf010-openshift-service-ca\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.483410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.483394 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-maas-api-tls\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.483550 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:26:45.483529 5561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-serving-cert: secret "maas-api-e2e-api-a-eb060a-serving-cert" not found Nov 02 20:26:45.483619 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:26:45.483607 5561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-maas-api-tls podName:43a5c044-6e8d-4c38-8665-809d63fdf010 nodeName:}" failed. No retries permitted until 2026-11-02 20:26:45.983585458 +0000 UTC m=+2042.791369653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-maas-api-tls") pod "maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" (UID: "43a5c044-6e8d-4c38-8665-809d63fdf010") : secret "maas-api-e2e-api-a-eb060a-serving-cert" not found Nov 02 20:26:45.483978 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.483958 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a5c044-6e8d-4c38-8665-809d63fdf010-openshift-service-ca\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.485596 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.485576 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-metrics-certs\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.537380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.537348 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmxjf\" (UniqueName: \"kubernetes.io/projected/43a5c044-6e8d-4c38-8665-809d63fdf010-kube-api-access-pmxjf\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.606822 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.606764 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:45.789652 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.789610 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:45.988995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.988955 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:45.991548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.991525 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-maas-api-tls\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:45.993959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:45.993939 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-maas-api-tls\") pod \"maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:46.285317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:46.285284 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:46.660900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:46.660854 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq"] Nov 02 20:26:46.667154 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:26:46.667119 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a5c044_6e8d_4c38_8665_809d63fdf010.slice/crio-9b0300207c287ec787194d37c53ce37da6c164aaece1d3fe79f5952a20fbcdc1 WatchSource:0}: Error finding container 9b0300207c287ec787194d37c53ce37da6c164aaece1d3fe79f5952a20fbcdc1: Status 404 returned error can't find the container with id 9b0300207c287ec787194d37c53ce37da6c164aaece1d3fe79f5952a20fbcdc1 Nov 02 20:26:47.561348 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:47.561315 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" event={"ID":"43a5c044-6e8d-4c38-8665-809d63fdf010","Type":"ContainerStarted","Data":"9b0300207c287ec787194d37c53ce37da6c164aaece1d3fe79f5952a20fbcdc1"} Nov 02 20:26:48.568706 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:48.568673 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" event={"ID":"43a5c044-6e8d-4c38-8665-809d63fdf010","Type":"ContainerStarted","Data":"e7321e6f93c318bf5c089f589d80c16fcde6cea95f0969b1ac44659d3ed85b77"} Nov 02 20:26:48.569111 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:48.568758 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:48.679560 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:48.679498 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" podStartSLOduration=2.839921444 podStartE2EDuration="3.679485258s" totalImagesPullingTime="839.563814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:48.676922349 +0000 UTC m=+2045.484706565" watchObservedRunningTime="2026-11-02 20:26:48.679485258 +0000 UTC m=+2045.487269474" Nov 02 20:26:50.326711 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:50.326676 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:52.570317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:52.570283 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:54.580464 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:54.580436 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:26:54.821588 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:54.821548 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:59.363381 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:26:59.363342 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:03.773464 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:03.773405 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:03.798147 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:27:03.798098 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:27:03.950780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:03.950747 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:04.176063 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:04.176029 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:08.563182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:08.563142 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:13.081935 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:13.081715 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:17.581719 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:17.581680 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:21.691529 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.691492 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk"] Nov 02 20:27:21.695980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.695959 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:21.748257 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.748227 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk"] Nov 02 20:27:21.769635 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.769602 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-metrics-certs\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:21.769801 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.769734 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn2hw\" (UniqueName: \"kubernetes.io/projected/daa3a147-4a02-49a7-8179-2045d8fd9489-kube-api-access-rn2hw\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:21.769909 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.769869 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa3a147-4a02-49a7-8179-2045d8fd9489-openshift-service-ca\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:21.769998 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.769986 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-maas-api-tls\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:21.792002 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.791971 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-602be312-serving-cert\"" Nov 02 20:27:21.871573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.871540 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa3a147-4a02-49a7-8179-2045d8fd9489-openshift-service-ca\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:21.871762 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.871628 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-maas-api-tls\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:21.871762 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.871693 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-metrics-certs\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:21.871919 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.871780 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn2hw\" (UniqueName: \"kubernetes.io/projected/daa3a147-4a02-49a7-8179-2045d8fd9489-kube-api-access-rn2hw\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:21.871919 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:27:21.871799 5561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-602be312-serving-cert: secret "maas-api-e2e-mt-602be312-serving-cert" not found Nov 02 20:27:21.871919 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:27:21.871912 5561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-maas-api-tls podName:daa3a147-4a02-49a7-8179-2045d8fd9489 nodeName:}" failed. No retries permitted until 2026-11-02 20:27:22.371861524 +0000 UTC m=+2079.179645719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-maas-api-tls") pod "maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" (UID: "daa3a147-4a02-49a7-8179-2045d8fd9489") : secret "maas-api-e2e-mt-602be312-serving-cert" not found Nov 02 20:27:21.872281 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.872259 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa3a147-4a02-49a7-8179-2045d8fd9489-openshift-service-ca\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:21.874101 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.874081 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-metrics-certs\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:21.947574 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.946870 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn2hw\" (UniqueName: \"kubernetes.io/projected/daa3a147-4a02-49a7-8179-2045d8fd9489-kube-api-access-rn2hw\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:21.947574 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:21.947420 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:22.123265 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:22.123215 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:22.296805 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:22.296723 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:22.381358 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:22.381325 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-maas-api-tls\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:22.383812 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:22.383786 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-maas-api-tls\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:22.609482 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:22.609397 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:22.983831 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:22.983799 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk"] Nov 02 20:27:22.990795 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:27:22.990764 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa3a147_4a02_49a7_8179_2045d8fd9489.slice/crio-abaa525b4f49c967760e372a114f485cf0763f4776b3df6084f7816f7ffacae4 WatchSource:0}: Error finding container abaa525b4f49c967760e372a114f485cf0763f4776b3df6084f7816f7ffacae4: Status 404 returned error can't find the container with id abaa525b4f49c967760e372a114f485cf0763f4776b3df6084f7816f7ffacae4 Nov 02 20:27:23.813674 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:23.813590 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" event={"ID":"daa3a147-4a02-49a7-8179-2045d8fd9489","Type":"ContainerStarted","Data":"abaa525b4f49c967760e372a114f485cf0763f4776b3df6084f7816f7ffacae4"} Nov 02 20:27:24.822056 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:24.822021 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" event={"ID":"daa3a147-4a02-49a7-8179-2045d8fd9489","Type":"ContainerStarted","Data":"69eb7ce38c5ad4ec4baccb3c011943dc22e6a740b2e9e11d134a2084012b6fc8"} Nov 02 20:27:24.822496 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:24.822120 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:24.932928 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:24.932864 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" podStartSLOduration=3.063799533 podStartE2EDuration="3.932849595s" totalImagesPullingTime="869.050062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:24.930539316 +0000 UTC m=+2081.738323551" watchObservedRunningTime="2026-11-02 20:27:24.932849595 +0000 UTC m=+2081.740633838" Nov 02 20:27:26.688243 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:26.688188 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:30.834559 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:30.834528 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:27:31.049193 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:31.049165 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:31.224937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:31.224377 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:31.397302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:31.397264 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:33.785113 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:27:33.785078 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:27:35.666011 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:35.665924 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:40.268546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:40.268507 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:43.226039 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:43.225995 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:43.416665 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:43.416629 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:45.007205 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:45.007152 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:49.881869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:49.881829 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:52.120132 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:52.120100 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:52.302632 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:52.302596 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:52.873858 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:52.873823 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:53.361328 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:53.361289 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:54.470120 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:54.469973 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:59.159010 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:27:59.158954 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:03.781068 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:28:03.781021 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:28:04.064594 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:04.064507 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:07.643563 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.643528 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5"] Nov 02 20:28:07.647491 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.647468 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.665627 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.665601 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5c2w\" (UniqueName: \"kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-kube-api-access-r5c2w\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.665782 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.665761 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df81b225-af07-4ecf-a502-2a85102530a1-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.665854 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.665801 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.665854 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.665822 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.666025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.666005 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.666097 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.666060 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.666154 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.666099 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.666199 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.666153 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.666302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.666280 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df81b225-af07-4ecf-a502-2a85102530a1-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.697306 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.697273 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-602be312-openshift-default-dockercfg-4c2qs\"" Nov 02 20:28:07.750121 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.750053 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5"] Nov 02 20:28:07.767754 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.767718 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df81b225-af07-4ecf-a502-2a85102530a1-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.767962 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.767807 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5c2w\" (UniqueName: \"kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-kube-api-access-r5c2w\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.767962 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.767951 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df81b225-af07-4ecf-a502-2a85102530a1-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.768099 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.767992 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.768099 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.768024 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.768211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.768096 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.768211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.768128 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.768211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.768180 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.768373 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.768224 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.768373 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.768342 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.768498 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.768471 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.768582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.768561 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.768645 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.768623 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.769374 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.769344 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df81b225-af07-4ecf-a502-2a85102530a1-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.770860 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.770835 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df81b225-af07-4ecf-a502-2a85102530a1-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.771119 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.771096 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.822078 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.822046 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.822250 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.822231 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5c2w\" (UniqueName: \"kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-kube-api-access-r5c2w\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:07.962071 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:07.962033 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:08.120894 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:08.120854 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z"] Nov 02 20:28:08.121146 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:08.121114 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" podUID="b39528d6-5e81-437f-b049-79e42cc4f6c4" containerName="istio-proxy" containerID="cri-o://c562c9b6e48e55f9f7acf2da9a1dacc818946397464f07b43f0456748395168e" gracePeriod=30 Nov 02 20:28:08.451311 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:08.451277 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5"] Nov 02 20:28:08.457568 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:28:08.457540 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf81b225_af07_4ecf_a502_2a85102530a1.slice/crio-cd3a4bd78795bfa2cdaf0f62be7287c58a64388c6e6a38ac9dca5ca53534d971 WatchSource:0}: Error finding container cd3a4bd78795bfa2cdaf0f62be7287c58a64388c6e6a38ac9dca5ca53534d971: Status 404 returned error can't find the container with id cd3a4bd78795bfa2cdaf0f62be7287c58a64388c6e6a38ac9dca5ca53534d971 Nov 02 20:28:08.460366 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:08.460333 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:28:08.460434 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:08.460399 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:28:08.460434 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:08.460429 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:28:08.791492 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:08.791385 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:08.958692 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:08.958658 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:28:08.958858 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:08.958712 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" podUID="b39528d6-5e81-437f-b049-79e42cc4f6c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:28:09.085262 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:09.085110 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:09.259144 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:09.259103 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" event={"ID":"df81b225-af07-4ecf-a502-2a85102530a1","Type":"ContainerStarted","Data":"abb8303282d0eefdf30225509862d7b7fee8f5b85a98f71f2f9a24a4f5de83f0"} Nov 02 20:28:09.259327 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:09.259162 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" event={"ID":"df81b225-af07-4ecf-a502-2a85102530a1","Type":"ContainerStarted","Data":"cd3a4bd78795bfa2cdaf0f62be7287c58a64388c6e6a38ac9dca5ca53534d971"} Nov 02 20:28:09.397664 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:09.397565 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" podStartSLOduration=2.397550237 podStartE2EDuration="2.397550237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:09.395551662 +0000 UTC m=+2126.203335877" watchObservedRunningTime="2026-11-02 20:28:09.397550237 +0000 UTC m=+2126.205334456" Nov 02 20:28:09.962205 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:09.962160 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:09.964002 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:09.963969 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:28:09.964101 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:09.964017 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" podUID="df81b225-af07-4ecf-a502-2a85102530a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 20:28:10.221410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:10.221324 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:10.500938 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:10.500843 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:10.962422 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:10.962375 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:28:10.963014 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:10.962435 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" podUID="df81b225-af07-4ecf-a502-2a85102530a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 20:28:11.962821 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:11.962788 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:28:11.963284 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:11.962839 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" podUID="df81b225-af07-4ecf-a502-2a85102530a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 20:28:12.893895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:12.884834 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:12.975903 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:12.971586 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:28:12.975903 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:12.971649 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" podUID="df81b225-af07-4ecf-a502-2a85102530a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 20:28:13.079730 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.079694 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:13.399458 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.398489 5561 generic.go:356] "Generic (PLEG): container finished" podID="b39528d6-5e81-437f-b049-79e42cc4f6c4" containerID="c562c9b6e48e55f9f7acf2da9a1dacc818946397464f07b43f0456748395168e" exitCode=0 Nov 02 20:28:13.399458 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.398617 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" event={"ID":"b39528d6-5e81-437f-b049-79e42cc4f6c4","Type":"ContainerDied","Data":"c562c9b6e48e55f9f7acf2da9a1dacc818946397464f07b43f0456748395168e"} Nov 02 20:28:13.493490 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.493463 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:28:13.671587 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.671543 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-data\") pod \"b39528d6-5e81-437f-b049-79e42cc4f6c4\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " Nov 02 20:28:13.671778 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.671646 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-podinfo\") pod \"b39528d6-5e81-437f-b049-79e42cc4f6c4\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " Nov 02 20:28:13.671778 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.671716 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-kube-api-access-m8w7q\" (UniqueName: \"kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-kube-api-access-m8w7q\") pod \"b39528d6-5e81-437f-b049-79e42cc4f6c4\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " Nov 02 20:28:13.671778 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.671751 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-credential-socket\") pod \"b39528d6-5e81-437f-b049-79e42cc4f6c4\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " Nov 02 20:28:13.671967 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.671795 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-certs\") pod \"b39528d6-5e81-437f-b049-79e42cc4f6c4\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " Nov 02 20:28:13.671967 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.671831 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39528d6-5e81-437f-b049-79e42cc4f6c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39528d6-5e81-437f-b049-79e42cc4f6c4-istiod-ca-cert\") pod \"b39528d6-5e81-437f-b049-79e42cc4f6c4\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " Nov 02 20:28:13.671967 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.671860 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-token\" (UniqueName: \"kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-token\") pod \"b39528d6-5e81-437f-b049-79e42cc4f6c4\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " Nov 02 20:28:13.672127 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.672048 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-socket\") pod \"b39528d6-5e81-437f-b049-79e42cc4f6c4\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " Nov 02 20:28:13.672177 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.672139 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-envoy\") pod \"b39528d6-5e81-437f-b049-79e42cc4f6c4\" (UID: \"b39528d6-5e81-437f-b049-79e42cc4f6c4\") " Nov 02 20:28:13.673959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.673490 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-credential-socket" pod "b39528d6-5e81-437f-b049-79e42cc4f6c4" (UID: "b39528d6-5e81-437f-b049-79e42cc4f6c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:13.673959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.673614 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-socket" pod "b39528d6-5e81-437f-b049-79e42cc4f6c4" (UID: "b39528d6-5e81-437f-b049-79e42cc4f6c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:13.673959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.673705 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-data" pod "b39528d6-5e81-437f-b049-79e42cc4f6c4" (UID: "b39528d6-5e81-437f-b049-79e42cc4f6c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:13.673959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.673716 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-certs" pod "b39528d6-5e81-437f-b049-79e42cc4f6c4" (UID: "b39528d6-5e81-437f-b049-79e42cc4f6c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:13.674624 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.674602 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39528d6-5e81-437f-b049-79e42cc4f6c4-istiod-ca-cert" pod "b39528d6-5e81-437f-b049-79e42cc4f6c4" (UID: "b39528d6-5e81-437f-b049-79e42cc4f6c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:13.677206 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.676196 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-envoy" pod "b39528d6-5e81-437f-b049-79e42cc4f6c4" (UID: "b39528d6-5e81-437f-b049-79e42cc4f6c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:13.678130 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.678093 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-kube-api-access-m8w7q" pod "b39528d6-5e81-437f-b049-79e42cc4f6c4" (UID: "b39528d6-5e81-437f-b049-79e42cc4f6c4"). InnerVolumeSpecName "kube-api-access-m8w7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:13.678336 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.678308 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-podinfo" pod "b39528d6-5e81-437f-b049-79e42cc4f6c4" (UID: "b39528d6-5e81-437f-b049-79e42cc4f6c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:28:13.678336 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.678315 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-token" pod "b39528d6-5e81-437f-b049-79e42cc4f6c4" (UID: "b39528d6-5e81-437f-b049-79e42cc4f6c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:13.776139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.776098 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:13.776139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.776138 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:13.776439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.776152 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:13.776439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.776165 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:13.776439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.776179 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8w7q\" (UniqueName: \"kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-kube-api-access-m8w7q\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:13.776439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.776193 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:13.776439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.776206 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39528d6-5e81-437f-b049-79e42cc4f6c4-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:13.776439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.776220 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39528d6-5e81-437f-b049-79e42cc4f6c4-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:13.776439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.776232 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39528d6-5e81-437f-b049-79e42cc4f6c4-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:13.962800 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.962706 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:28:13.963078 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:13.963043 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" podUID="df81b225-af07-4ecf-a502-2a85102530a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 20:28:14.128898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:14.128431 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:14.412289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:14.412257 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" Nov 02 20:28:14.412479 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:14.412391 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z" event={"ID":"b39528d6-5e81-437f-b049-79e42cc4f6c4","Type":"ContainerDied","Data":"26618d649b6dc715ccf2987363ff9f7a8ecc2cafe5fa2bab8c51b7b16562231f"} Nov 02 20:28:14.412479 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:14.412458 5561 scope.go:122] "RemoveContainer" containerID="c562c9b6e48e55f9f7acf2da9a1dacc818946397464f07b43f0456748395168e" Nov 02 20:28:14.622798 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:14.622763 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z"] Nov 02 20:28:14.670475 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:14.670392 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-wpt7z"] Nov 02 20:28:15.034584 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:15.034509 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:15.034922 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:15.034902 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:15.035867 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:15.035842 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:28:15.318255 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:15.318173 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39528d6-5e81-437f-b049-79e42cc4f6c4" path="/var/lib/kubelet/pods/b39528d6-5e81-437f-b049-79e42cc4f6c4/volumes" Nov 02 20:28:19.154698 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:19.150813 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:22.765638 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.765598 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p"] Nov 02 20:28:22.766508 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.766491 5561 cpu_manager.go:414] "containerMap: removing container" podUID="b39528d6-5e81-437f-b049-79e42cc4f6c4" containerName="istio-proxy" Nov 02 20:28:22.766559 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.766511 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39528d6-5e81-437f-b049-79e42cc4f6c4" containerName="istio-proxy" Nov 02 20:28:22.766682 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.766672 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39528d6-5e81-437f-b049-79e42cc4f6c4" containerName="istio-proxy" Nov 02 20:28:22.772060 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.772037 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.835134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.835093 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.835134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.835133 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfbk4\" (UniqueName: \"kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-kube-api-access-cfbk4\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.835431 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.835230 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.835431 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.835305 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.835431 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.835372 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.835560 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.835483 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.835601 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.835583 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.835642 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.835612 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1487346-6dd4-4ea6-8507-bc97491d3d12-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.835744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.835731 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.936720 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.936678 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.936911 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.936774 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.936911 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.936844 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.937036 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.936974 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.937036 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.937003 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1487346-6dd4-4ea6-8507-bc97491d3d12-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.937149 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.937122 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.937220 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.937188 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.937220 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.937214 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfbk4\" (UniqueName: \"kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-kube-api-access-cfbk4\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.937321 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.937258 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.937421 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.937399 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.937564 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.937456 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.937940 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.937916 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.938067 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.938002 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.938067 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.938050 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1487346-6dd4-4ea6-8507-bc97491d3d12-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.939994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.939970 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.944122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.944096 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.968057 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.968019 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p"] Nov 02 20:28:22.991320 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.991286 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:22.991716 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:22.991694 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfbk4\" (UniqueName: \"kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-kube-api-access-cfbk4\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:23.085652 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:23.085560 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:23.521047 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:23.521017 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p"] Nov 02 20:28:23.523065 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:28:23.523030 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1487346_6dd4_4ea6_8507_bc97491d3d12.slice/crio-b654c72bbfeaff7d39389554f27e5d2df1b840e31794201f08f3d0977b4dd4b4 WatchSource:0}: Error finding container b654c72bbfeaff7d39389554f27e5d2df1b840e31794201f08f3d0977b4dd4b4: Status 404 returned error can't find the container with id b654c72bbfeaff7d39389554f27e5d2df1b840e31794201f08f3d0977b4dd4b4 Nov 02 20:28:23.530277 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:23.530241 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:28:23.530362 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:23.530313 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:28:23.530362 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:23.530342 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:28:24.012957 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:24.012925 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:24.481861 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:24.481815 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" event={"ID":"f1487346-6dd4-4ea6-8507-bc97491d3d12","Type":"ContainerStarted","Data":"ef62c1d7e439bf561b33c7d2de9d9ade5e08e76ace63957e8f8a7aa70a76d5a2"} Nov 02 20:28:24.482088 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:24.481918 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" event={"ID":"f1487346-6dd4-4ea6-8507-bc97491d3d12","Type":"ContainerStarted","Data":"b654c72bbfeaff7d39389554f27e5d2df1b840e31794201f08f3d0977b4dd4b4"} Nov 02 20:28:25.085972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:25.085937 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:25.087221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:25.087194 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:28:25.087326 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:25.087244 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" podUID="f1487346-6dd4-4ea6-8507-bc97491d3d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 20:28:26.086694 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:26.086652 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:28:26.087141 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:26.086724 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" podUID="f1487346-6dd4-4ea6-8507-bc97491d3d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 20:28:27.086620 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:27.086584 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:28:27.086800 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:27.086653 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" podUID="f1487346-6dd4-4ea6-8507-bc97491d3d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 20:28:27.617135 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:27.617100 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:27.800574 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:27.800541 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:28.086722 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:28.086681 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:28:28.086941 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:28.086740 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" podUID="f1487346-6dd4-4ea6-8507-bc97491d3d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 20:28:28.721725 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:28.721685 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:28.979181 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:28.978956 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:29.086405 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:29.086368 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:28:29.086579 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:29.086433 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" podUID="f1487346-6dd4-4ea6-8507-bc97491d3d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 20:28:30.086681 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:30.086625 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:28:30.087499 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:30.086690 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" podUID="f1487346-6dd4-4ea6-8507-bc97491d3d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 20:28:30.284363 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:30.284309 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:30.991781 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:30.991746 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:31.091034 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:31.091001 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:31.091457 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:31.091429 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:31.092159 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:31.092140 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:28:31.261924 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:31.261802 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" podStartSLOduration=9.261788487 podStartE2EDuration="9.261788487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:24.644179994 +0000 UTC m=+2141.451964228" watchObservedRunningTime="2026-11-02 20:28:31.261788487 +0000 UTC m=+2148.069572703" Nov 02 20:28:31.438252 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:31.438218 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:31.616922 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:31.616814 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:33.210369 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.210332 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq"] Nov 02 20:28:33.211154 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.211117 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" podUID="43a5c044-6e8d-4c38-8665-809d63fdf010" containerName="maas-api" containerID="cri-o://e7321e6f93c318bf5c089f589d80c16fcde6cea95f0969b1ac44659d3ed85b77" gracePeriod=30 Nov 02 20:28:33.276854 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.276826 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:33.452070 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.452038 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:33.701625 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.695707 5561 generic.go:356] "Generic (PLEG): container finished" podID="43a5c044-6e8d-4c38-8665-809d63fdf010" containerID="e7321e6f93c318bf5c089f589d80c16fcde6cea95f0969b1ac44659d3ed85b77" exitCode=0 Nov 02 20:28:33.701625 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.695937 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" event={"ID":"43a5c044-6e8d-4c38-8665-809d63fdf010","Type":"ContainerDied","Data":"e7321e6f93c318bf5c089f589d80c16fcde6cea95f0969b1ac44659d3ed85b77"} Nov 02 20:28:33.787919 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.787855 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:28:33.796391 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:28:33.796307 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:28:33.842829 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.842787 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-maas-api-tls\") pod \"43a5c044-6e8d-4c38-8665-809d63fdf010\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " Nov 02 20:28:33.843041 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.842928 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a5c044-6e8d-4c38-8665-809d63fdf010-kube-api-access-pmxjf\" (UniqueName: \"kubernetes.io/projected/43a5c044-6e8d-4c38-8665-809d63fdf010-kube-api-access-pmxjf\") pod \"43a5c044-6e8d-4c38-8665-809d63fdf010\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " Nov 02 20:28:33.843041 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.842985 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-metrics-certs\") pod \"43a5c044-6e8d-4c38-8665-809d63fdf010\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " Nov 02 20:28:33.843263 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.843234 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a5c044-6e8d-4c38-8665-809d63fdf010-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a5c044-6e8d-4c38-8665-809d63fdf010-openshift-service-ca\") pod \"43a5c044-6e8d-4c38-8665-809d63fdf010\" (UID: \"43a5c044-6e8d-4c38-8665-809d63fdf010\") " Nov 02 20:28:33.843598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.843572 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a5c044-6e8d-4c38-8665-809d63fdf010-openshift-service-ca" pod "43a5c044-6e8d-4c38-8665-809d63fdf010" (UID: "43a5c044-6e8d-4c38-8665-809d63fdf010"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:33.843868 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.843851 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a5c044-6e8d-4c38-8665-809d63fdf010-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:33.845063 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.845044 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-metrics-certs" pod "43a5c044-6e8d-4c38-8665-809d63fdf010" (UID: "43a5c044-6e8d-4c38-8665-809d63fdf010"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:28:33.845462 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.845434 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-maas-api-tls" pod "43a5c044-6e8d-4c38-8665-809d63fdf010" (UID: "43a5c044-6e8d-4c38-8665-809d63fdf010"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:28:33.845539 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.845475 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a5c044-6e8d-4c38-8665-809d63fdf010-kube-api-access-pmxjf" pod "43a5c044-6e8d-4c38-8665-809d63fdf010" (UID: "43a5c044-6e8d-4c38-8665-809d63fdf010"). InnerVolumeSpecName "kube-api-access-pmxjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:33.945616 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.945567 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:33.945616 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.945619 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmxjf\" (UniqueName: \"kubernetes.io/projected/43a5c044-6e8d-4c38-8665-809d63fdf010-kube-api-access-pmxjf\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:33.945860 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:33.945637 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a5c044-6e8d-4c38-8665-809d63fdf010-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:34.702964 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:34.702938 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" Nov 02 20:28:34.703395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:34.702935 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq" event={"ID":"43a5c044-6e8d-4c38-8665-809d63fdf010","Type":"ContainerDied","Data":"9b0300207c287ec787194d37c53ce37da6c164aaece1d3fe79f5952a20fbcdc1"} Nov 02 20:28:34.703395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:34.703146 5561 scope.go:122] "RemoveContainer" containerID="e7321e6f93c318bf5c089f589d80c16fcde6cea95f0969b1ac44659d3ed85b77" Nov 02 20:28:34.892116 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:34.892083 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq"] Nov 02 20:28:34.896991 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:34.896963 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eb060a-7444b4b655-tqqvq"] Nov 02 20:28:35.074575 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:35.074425 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:35.314202 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:35.314169 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a5c044-6e8d-4c38-8665-809d63fdf010" path="/var/lib/kubelet/pods/43a5c044-6e8d-4c38-8665-809d63fdf010/volumes" Nov 02 20:28:37.890791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:37.890755 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p"] Nov 02 20:28:37.891358 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:37.891344 5561 cpu_manager.go:414] "containerMap: removing container" podUID="43a5c044-6e8d-4c38-8665-809d63fdf010" containerName="maas-api" Nov 02 20:28:37.891407 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:37.891358 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a5c044-6e8d-4c38-8665-809d63fdf010" containerName="maas-api" Nov 02 20:28:37.891462 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:37.891453 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a5c044-6e8d-4c38-8665-809d63fdf010" containerName="maas-api" Nov 02 20:28:37.895433 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:37.895416 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:37.975885 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:37.975840 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p"] Nov 02 20:28:38.003058 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.003026 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.003257 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.003089 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.003257 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.003109 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p655t\" (UniqueName: \"kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-kube-api-access-p655t\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.003389 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.003363 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.003495 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.003474 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.003589 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.003567 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.003684 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.003665 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.003731 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.003721 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a574dd-f668-47bc-8ac4-8b11316a1a11-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.003780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.003743 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.105474 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.105427 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a574dd-f668-47bc-8ac4-8b11316a1a11-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.105474 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.105480 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.105721 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.105503 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.105721 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.105550 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.105721 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.105571 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p655t\" (UniqueName: \"kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-kube-api-access-p655t\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.105721 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.105700 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.106011 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.105777 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.106011 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.105832 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.106011 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.105936 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.106178 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.106159 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.106291 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.106265 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.106353 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.106284 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.106406 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.106369 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.106468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.106395 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a574dd-f668-47bc-8ac4-8b11316a1a11-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.108117 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.108086 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.108325 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.108302 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.159725 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.159696 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.159891 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.159780 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p655t\" (UniqueName: \"kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-kube-api-access-p655t\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-pq56p\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.209465 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.209424 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:38.645021 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.644935 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p"] Nov 02 20:28:38.650951 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:28:38.650922 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a574dd_f668_47bc_8ac4_8b11316a1a11.slice/crio-d7ffb8ac2fe64d4fc4a43edf6d9276b0388d0c4141f70521eb843fdbfd480f29 WatchSource:0}: Error finding container d7ffb8ac2fe64d4fc4a43edf6d9276b0388d0c4141f70521eb843fdbfd480f29: Status 404 returned error can't find the container with id d7ffb8ac2fe64d4fc4a43edf6d9276b0388d0c4141f70521eb843fdbfd480f29 Nov 02 20:28:38.653729 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.653690 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:28:38.653831 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.653772 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:28:38.653831 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.653808 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:28:38.733408 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.733368 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" event={"ID":"53a574dd-f668-47bc-8ac4-8b11316a1a11","Type":"ContainerStarted","Data":"050f1176158376dc866cc094b38f4dfc5a114ad66aa77c06aeaa4565517579f6"} Nov 02 20:28:38.733773 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.733421 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" event={"ID":"53a574dd-f668-47bc-8ac4-8b11316a1a11","Type":"ContainerStarted","Data":"d7ffb8ac2fe64d4fc4a43edf6d9276b0388d0c4141f70521eb843fdbfd480f29"} Nov 02 20:28:38.847094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:38.847041 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" podStartSLOduration=1.84702529 podStartE2EDuration="1.84702529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:38.843997694 +0000 UTC m=+2155.651781927" watchObservedRunningTime="2026-11-02 20:28:38.84702529 +0000 UTC m=+2155.654809505" Nov 02 20:28:39.210273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:39.210237 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:39.212070 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:39.212046 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 20:28:39.212184 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:39.212102 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" podUID="53a574dd-f668-47bc-8ac4-8b11316a1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 20:28:40.033897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:40.029808 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:40.210563 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:40.210521 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 20:28:40.211089 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:40.210582 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" podUID="53a574dd-f668-47bc-8ac4-8b11316a1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 20:28:41.209807 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:41.209774 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 20:28:41.210045 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:41.209831 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" podUID="53a574dd-f668-47bc-8ac4-8b11316a1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 20:28:41.664713 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:41.664671 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:41.850721 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:41.850669 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:42.210527 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:42.210490 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 20:28:42.210709 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:42.210563 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" podUID="53a574dd-f668-47bc-8ac4-8b11316a1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 20:28:43.211925 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:43.211512 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 20:28:43.211925 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:43.211582 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" podUID="53a574dd-f668-47bc-8ac4-8b11316a1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 20:28:44.210496 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:44.210462 5561 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 20:28:44.210673 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:44.210526 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" podUID="53a574dd-f668-47bc-8ac4-8b11316a1a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 20:28:45.000540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:45.000459 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:45.295238 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:45.295148 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:45.295792 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:45.295733 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:45.296397 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:45.296376 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:28:49.132017 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:49.131977 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:50.671497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:50.671463 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:50.937334 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:50.937255 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:51.127793 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:51.127763 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:53.233806 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:53.233768 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:53.752987 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:53.752937 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:56.019324 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:28:56.019285 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:00.789246 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:00.789163 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:01.107027 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:01.106925 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq"] Nov 02 20:29:01.476585 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:01.476544 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:01.656287 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:01.656256 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:01.831148 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:01.831063 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:03.779522 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:29:03.779486 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:29:05.577212 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:05.577156 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:06.107380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.107332 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" podUID="ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a" containerName="payload-pre-processing" containerID="cri-o://2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2" gracePeriod=25 Nov 02 20:29:06.380629 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.380605 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" Nov 02 20:29:06.494512 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.494475 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-plugins-config-volume\") pod \"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a\" (UID: \"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a\") " Nov 02 20:29:06.494673 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.494655 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-kube-api-access-2d57d\" (UniqueName: \"kubernetes.io/projected/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-kube-api-access-2d57d\") pod \"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a\" (UID: \"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a\") " Nov 02 20:29:06.494934 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.494907 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-plugins-config-volume" pod "ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a" (UID: "ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:06.495386 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.495365 5561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-plugins-config-volume\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:06.496907 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.496860 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-kube-api-access-2d57d" pod "ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a" (UID: "ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a"). InnerVolumeSpecName "kube-api-access-2d57d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:06.597258 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.597222 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d57d\" (UniqueName: \"kubernetes.io/projected/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a-kube-api-access-2d57d\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:06.927599 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.927565 5561 generic.go:356] "Generic (PLEG): container finished" podID="ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a" containerID="2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2" exitCode=0 Nov 02 20:29:06.927778 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.927614 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" Nov 02 20:29:06.927778 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.927650 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" event={"ID":"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a","Type":"ContainerDied","Data":"2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2"} Nov 02 20:29:06.927778 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.927704 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq" event={"ID":"ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a","Type":"ContainerDied","Data":"c9b4479ad3a71b07bfeded66ed9a286df88d67f93ae651292156f9e71e718153"} Nov 02 20:29:06.927778 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.927732 5561 scope.go:122] "RemoveContainer" containerID="2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2" Nov 02 20:29:06.949375 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.949352 5561 scope.go:122] "RemoveContainer" containerID="2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2" Nov 02 20:29:06.949693 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:29:06.949675 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2\": container with ID starting with 2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2 not found: ID does not exist" containerID="2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2" Nov 02 20:29:06.949758 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:06.949703 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2"} err="failed to get container status \"2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2\": rpc error: code = NotFound desc = could not find container \"2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2\": container with ID starting with 2145c4e32adeede1fe74b3fc62fd87e9f5ea7eeec15aee122c63d3002bf386e2 not found: ID does not exist" Nov 02 20:29:07.084582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:07.084547 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq"] Nov 02 20:29:07.131803 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:07.131766 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abc937-7f79cf45f-nwjkq"] Nov 02 20:29:07.314449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:07.314373 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a" path="/var/lib/kubelet/pods/ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a/volumes" Nov 02 20:29:07.434312 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:07.434279 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:07.614706 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:07.614633 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:07.812106 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:07.812070 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:09.877788 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:09.877753 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:10.268901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:10.266032 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:10.947899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:10.944926 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl"] Nov 02 20:29:11.056583 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:11.054199 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:11.058191 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:11.058161 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4"] Nov 02 20:29:11.276724 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:11.276548 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:11.489934 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:11.489340 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:11.668567 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:11.668526 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:14.344121 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:14.344075 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:14.943533 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:14.943500 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:15.135048 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:15.135014 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:15.945461 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:15.945392 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" podUID="6e8276a3-e24b-4a21-8522-c5bac20d5a3b" containerName="payload-processing" containerID="cri-o://3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a" gracePeriod=25 Nov 02 20:29:15.965739 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:15.965704 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:16.059482 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.059422 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" podUID="b6051c83-da4e-482c-8833-ca0dd2bc9bcf" containerName="payload-pre-processing" containerID="cri-o://72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47" gracePeriod=25 Nov 02 20:29:16.223726 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.223692 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" Nov 02 20:29:16.234307 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.234276 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:16.306936 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.293786 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-plugins-config-volume\") pod \"6e8276a3-e24b-4a21-8522-c5bac20d5a3b\" (UID: \"6e8276a3-e24b-4a21-8522-c5bac20d5a3b\") " Nov 02 20:29:16.306936 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.293996 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-kube-api-access-ck96g\" (UniqueName: \"kubernetes.io/projected/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-kube-api-access-ck96g\") pod \"6e8276a3-e24b-4a21-8522-c5bac20d5a3b\" (UID: \"6e8276a3-e24b-4a21-8522-c5bac20d5a3b\") " Nov 02 20:29:16.306936 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.297659 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-plugins-config-volume" pod "6e8276a3-e24b-4a21-8522-c5bac20d5a3b" (UID: "6e8276a3-e24b-4a21-8522-c5bac20d5a3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:16.318926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.314395 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-kube-api-access-ck96g" pod "6e8276a3-e24b-4a21-8522-c5bac20d5a3b" (UID: "6e8276a3-e24b-4a21-8522-c5bac20d5a3b"). InnerVolumeSpecName "kube-api-access-ck96g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:16.371664 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.371638 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" Nov 02 20:29:16.397107 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.397072 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-kube-api-access-7wt2r\" (UniqueName: \"kubernetes.io/projected/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-kube-api-access-7wt2r\") pod \"b6051c83-da4e-482c-8833-ca0dd2bc9bcf\" (UID: \"b6051c83-da4e-482c-8833-ca0dd2bc9bcf\") " Nov 02 20:29:16.397251 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.397184 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-plugins-config-volume\") pod \"b6051c83-da4e-482c-8833-ca0dd2bc9bcf\" (UID: \"b6051c83-da4e-482c-8833-ca0dd2bc9bcf\") " Nov 02 20:29:16.397617 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.397588 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-plugins-config-volume" pod "b6051c83-da4e-482c-8833-ca0dd2bc9bcf" (UID: "b6051c83-da4e-482c-8833-ca0dd2bc9bcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:16.397794 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.397776 5561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-plugins-config-volume\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:16.397865 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.397799 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck96g\" (UniqueName: \"kubernetes.io/projected/6e8276a3-e24b-4a21-8522-c5bac20d5a3b-kube-api-access-ck96g\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:16.397865 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.397813 5561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-plugins-config-volume\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:16.399279 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.399260 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-kube-api-access-7wt2r" pod "b6051c83-da4e-482c-8833-ca0dd2bc9bcf" (UID: "b6051c83-da4e-482c-8833-ca0dd2bc9bcf"). InnerVolumeSpecName "kube-api-access-7wt2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:16.418973 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.418943 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:16.500164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.500004 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wt2r\" (UniqueName: \"kubernetes.io/projected/b6051c83-da4e-482c-8833-ca0dd2bc9bcf-kube-api-access-7wt2r\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:16.851869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:16.851790 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:17.040179 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.040148 5561 generic.go:356] "Generic (PLEG): container finished" podID="6e8276a3-e24b-4a21-8522-c5bac20d5a3b" containerID="3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a" exitCode=0 Nov 02 20:29:17.040603 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.040196 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" Nov 02 20:29:17.040603 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.040224 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" event={"ID":"6e8276a3-e24b-4a21-8522-c5bac20d5a3b","Type":"ContainerDied","Data":"3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a"} Nov 02 20:29:17.040603 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.040272 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl" event={"ID":"6e8276a3-e24b-4a21-8522-c5bac20d5a3b","Type":"ContainerDied","Data":"c5b3da5686966a07a8aa3f01ba61ed8dd9421765301e78796c230c13257a7b99"} Nov 02 20:29:17.040603 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.040297 5561 scope.go:122] "RemoveContainer" containerID="3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a" Nov 02 20:29:17.041813 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.041793 5561 generic.go:356] "Generic (PLEG): container finished" podID="b6051c83-da4e-482c-8833-ca0dd2bc9bcf" containerID="72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47" exitCode=0 Nov 02 20:29:17.041932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.041838 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" Nov 02 20:29:17.041932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.041849 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" event={"ID":"b6051c83-da4e-482c-8833-ca0dd2bc9bcf","Type":"ContainerDied","Data":"72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47"} Nov 02 20:29:17.041932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.041904 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4" event={"ID":"b6051c83-da4e-482c-8833-ca0dd2bc9bcf","Type":"ContainerDied","Data":"12638880d890891e40a77e521dd0f0ec866b70ff7992946274128961ce5f554b"} Nov 02 20:29:17.052284 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.052207 5561 scope.go:122] "RemoveContainer" containerID="3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a" Nov 02 20:29:17.052542 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:29:17.052517 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a\": container with ID starting with 3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a not found: ID does not exist" containerID="3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a" Nov 02 20:29:17.052627 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.052554 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a"} err="failed to get container status \"3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a\": rpc error: code = NotFound desc = could not find container \"3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a\": container with ID starting with 3620de714c37b7c5341f21111cfa1a1485c1fcf5671156318f997a56fd405b7a not found: ID does not exist" Nov 02 20:29:17.052627 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.052579 5561 scope.go:122] "RemoveContainer" containerID="72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47" Nov 02 20:29:17.062130 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.062108 5561 scope.go:122] "RemoveContainer" containerID="72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47" Nov 02 20:29:17.062456 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:29:17.062435 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47\": container with ID starting with 72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47 not found: ID does not exist" containerID="72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47" Nov 02 20:29:17.062506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.062466 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47"} err="failed to get container status \"72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47\": rpc error: code = NotFound desc = could not find container \"72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47\": container with ID starting with 72b83ae7d17fe26da3c9a98f4a37f46e0e8c7a848648c562ef8d9e93559a7f47 not found: ID does not exist" Nov 02 20:29:17.200008 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.199966 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4"] Nov 02 20:29:17.203750 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.203180 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-602be312-9978fb9cf-r4kf4"] Nov 02 20:29:17.315364 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.315334 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6051c83-da4e-482c-8833-ca0dd2bc9bcf" path="/var/lib/kubelet/pods/b6051c83-da4e-482c-8833-ca0dd2bc9bcf/volumes" Nov 02 20:29:17.354974 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.354938 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl"] Nov 02 20:29:17.357772 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:17.357747 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-602be312-56bf49fbb9-7rvhl"] Nov 02 20:29:18.447904 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:18.447830 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:18.627479 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:18.624952 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:19.314456 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:19.314427 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e8276a3-e24b-4a21-8522-c5bac20d5a3b" path="/var/lib/kubelet/pods/6e8276a3-e24b-4a21-8522-c5bac20d5a3b/volumes" Nov 02 20:29:21.360614 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:21.349457 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:22.666946 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:22.666907 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:22.956053 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:22.955965 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:23.194613 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:23.180350 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:26.511916 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:26.501148 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:31.290486 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:31.289631 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:33.780361 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:29:33.780323 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:29:35.118715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:35.118672 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:35.552825 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:35.552783 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:35.725523 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:35.725485 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:35.900784 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:35.900705 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:37.048338 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:37.048301 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:37.250140 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:37.250103 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:37.468556 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:37.468523 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:37.919562 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:37.919520 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:39.666953 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:39.666912 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:39.861558 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:39.861522 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:41.551869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:41.551829 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:43.037779 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:43.037746 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:43.250742 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:43.250691 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:43.430252 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:43.430215 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:43.759106 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:43.759018 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:44.085421 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:44.085335 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:44.581782 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:44.581724 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:44.754723 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:44.754677 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:44.884318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:44.884225 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9"] Nov 02 20:29:44.884494 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:44.884429 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" podUID="c29c14fb-7305-431d-ad95-e51e4ea1b417" containerName="istio-proxy" containerID="cri-o://33d492eb5403cc48f9d844a19627396134dd6cad4f1f0b12e16db2b2f272e481" gracePeriod=30 Nov 02 20:29:44.887454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:44.887428 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84"] Nov 02 20:29:44.887672 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:44.887637 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" podUID="8f544e1a-31c2-4601-a0d3-95f214fa213d" containerName="istio-proxy" containerID="cri-o://49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1" gracePeriod=30 Nov 02 20:29:44.959056 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:44.959022 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:48.323782 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:48.323742 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:49.659700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:49.659666 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:49.843937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:49.843905 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:50.072953 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.072839 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:50.337912 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.337862 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:29:50.345893 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.345840 5561 generic.go:356] "Generic (PLEG): container finished" podID="8f544e1a-31c2-4601-a0d3-95f214fa213d" containerID="49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1" exitCode=0 Nov 02 20:29:50.346040 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.345917 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" Nov 02 20:29:50.346102 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.346044 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" event={"ID":"8f544e1a-31c2-4601-a0d3-95f214fa213d","Type":"ContainerDied","Data":"49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1"} Nov 02 20:29:50.346152 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.346121 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84" event={"ID":"8f544e1a-31c2-4601-a0d3-95f214fa213d","Type":"ContainerDied","Data":"16508a965bcdf505d1e2b983a7820047eaad457d0909a86a29c77d114dce7a34"} Nov 02 20:29:50.346208 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.346154 5561 scope.go:122] "RemoveContainer" containerID="49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1" Nov 02 20:29:50.348464 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.348438 5561 generic.go:356] "Generic (PLEG): container finished" podID="c29c14fb-7305-431d-ad95-e51e4ea1b417" containerID="33d492eb5403cc48f9d844a19627396134dd6cad4f1f0b12e16db2b2f272e481" exitCode=0 Nov 02 20:29:50.348563 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.348520 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" event={"ID":"c29c14fb-7305-431d-ad95-e51e4ea1b417","Type":"ContainerDied","Data":"33d492eb5403cc48f9d844a19627396134dd6cad4f1f0b12e16db2b2f272e481"} Nov 02 20:29:50.359598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.359167 5561 scope.go:122] "RemoveContainer" containerID="49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1" Nov 02 20:29:50.362469 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:29:50.362439 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1\": container with ID starting with 49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1 not found: ID does not exist" containerID="49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1" Nov 02 20:29:50.362578 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.362478 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1"} err="failed to get container status \"49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1\": rpc error: code = NotFound desc = could not find container \"49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1\": container with ID starting with 49b09a1b2643db32f003a8c0035f1a4b0c90457b3f4f22b80d7b1f8dc21de1c1 not found: ID does not exist" Nov 02 20:29:50.391384 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.391359 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:29:50.499380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499342 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-token\" (UniqueName: \"kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-token\") pod \"c29c14fb-7305-431d-ad95-e51e4ea1b417\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " Nov 02 20:29:50.499578 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499442 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-envoy\") pod \"8f544e1a-31c2-4601-a0d3-95f214fa213d\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " Nov 02 20:29:50.499578 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499563 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-certs\") pod \"8f544e1a-31c2-4601-a0d3-95f214fa213d\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " Nov 02 20:29:50.499705 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499617 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-data\") pod \"c29c14fb-7305-431d-ad95-e51e4ea1b417\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " Nov 02 20:29:50.499705 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499659 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-podinfo\") pod \"c29c14fb-7305-431d-ad95-e51e4ea1b417\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " Nov 02 20:29:50.499705 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499689 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-podinfo\") pod \"8f544e1a-31c2-4601-a0d3-95f214fa213d\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499730 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-data\") pod \"8f544e1a-31c2-4601-a0d3-95f214fa213d\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499764 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-socket\") pod \"8f544e1a-31c2-4601-a0d3-95f214fa213d\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499802 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-envoy\") pod \"c29c14fb-7305-431d-ad95-e51e4ea1b417\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499812 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-certs" pod "8f544e1a-31c2-4601-a0d3-95f214fa213d" (UID: "8f544e1a-31c2-4601-a0d3-95f214fa213d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499918 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f544e1a-31c2-4601-a0d3-95f214fa213d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f544e1a-31c2-4601-a0d3-95f214fa213d-istiod-ca-cert\") pod \"8f544e1a-31c2-4601-a0d3-95f214fa213d\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499961 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-data" pod "c29c14fb-7305-431d-ad95-e51e4ea1b417" (UID: "c29c14fb-7305-431d-ad95-e51e4ea1b417"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.499971 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-token\" (UniqueName: \"kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-token\") pod \"8f544e1a-31c2-4601-a0d3-95f214fa213d\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.500097 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-kube-api-access-7tq7v\" (UniqueName: \"kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-kube-api-access-7tq7v\") pod \"c29c14fb-7305-431d-ad95-e51e4ea1b417\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.500207 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-socket" pod "8f544e1a-31c2-4601-a0d3-95f214fa213d" (UID: "8f544e1a-31c2-4601-a0d3-95f214fa213d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.500224 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-data" pod "8f544e1a-31c2-4601-a0d3-95f214fa213d" (UID: "8f544e1a-31c2-4601-a0d3-95f214fa213d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.500231 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-credential-socket\") pod \"8f544e1a-31c2-4601-a0d3-95f214fa213d\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.500298 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-socket\") pod \"c29c14fb-7305-431d-ad95-e51e4ea1b417\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.500397 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-credential-socket\") pod \"c29c14fb-7305-431d-ad95-e51e4ea1b417\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.500434 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29c14fb-7305-431d-ad95-e51e4ea1b417-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c14fb-7305-431d-ad95-e51e4ea1b417-istiod-ca-cert\") pod \"c29c14fb-7305-431d-ad95-e51e4ea1b417\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.500488 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-credential-socket" pod "8f544e1a-31c2-4601-a0d3-95f214fa213d" (UID: "8f544e1a-31c2-4601-a0d3-95f214fa213d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.500611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.500526 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f544e1a-31c2-4601-a0d3-95f214fa213d-istiod-ca-cert" pod "8f544e1a-31c2-4601-a0d3-95f214fa213d" (UID: "8f544e1a-31c2-4601-a0d3-95f214fa213d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:50.501764 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.501131 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-credential-socket" pod "c29c14fb-7305-431d-ad95-e51e4ea1b417" (UID: "c29c14fb-7305-431d-ad95-e51e4ea1b417"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.501764 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.501454 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-socket" pod "c29c14fb-7305-431d-ad95-e51e4ea1b417" (UID: "c29c14fb-7305-431d-ad95-e51e4ea1b417"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.501764 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.501642 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-certs\") pod \"c29c14fb-7305-431d-ad95-e51e4ea1b417\" (UID: \"c29c14fb-7305-431d-ad95-e51e4ea1b417\") " Nov 02 20:29:50.501764 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.501686 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-kube-api-access-4rxkb\" (UniqueName: \"kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-kube-api-access-4rxkb\") pod \"8f544e1a-31c2-4601-a0d3-95f214fa213d\" (UID: \"8f544e1a-31c2-4601-a0d3-95f214fa213d\") " Nov 02 20:29:50.501997 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.501908 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-certs" pod "c29c14fb-7305-431d-ad95-e51e4ea1b417" (UID: "c29c14fb-7305-431d-ad95-e51e4ea1b417"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.502385 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.502364 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.502385 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.502385 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.502546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.502394 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.502546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.502402 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.502546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.502410 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.502546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.502419 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f544e1a-31c2-4601-a0d3-95f214fa213d-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.502546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.502427 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.502546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.502435 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.502546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.502443 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.502947 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.502629 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-token" pod "c29c14fb-7305-431d-ad95-e51e4ea1b417" (UID: "c29c14fb-7305-431d-ad95-e51e4ea1b417"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:50.503843 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.503811 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-podinfo" pod "8f544e1a-31c2-4601-a0d3-95f214fa213d" (UID: "8f544e1a-31c2-4601-a0d3-95f214fa213d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:29:50.503976 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.503922 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-podinfo" pod "c29c14fb-7305-431d-ad95-e51e4ea1b417" (UID: "c29c14fb-7305-431d-ad95-e51e4ea1b417"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:29:50.504318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.504295 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-kube-api-access-4rxkb" pod "8f544e1a-31c2-4601-a0d3-95f214fa213d" (UID: "8f544e1a-31c2-4601-a0d3-95f214fa213d"). InnerVolumeSpecName "kube-api-access-4rxkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:50.504988 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.504962 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-envoy" pod "8f544e1a-31c2-4601-a0d3-95f214fa213d" (UID: "8f544e1a-31c2-4601-a0d3-95f214fa213d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.504988 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.504981 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-token" pod "8f544e1a-31c2-4601-a0d3-95f214fa213d" (UID: "8f544e1a-31c2-4601-a0d3-95f214fa213d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:50.505158 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.505137 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-kube-api-access-7tq7v" pod "c29c14fb-7305-431d-ad95-e51e4ea1b417" (UID: "c29c14fb-7305-431d-ad95-e51e4ea1b417"). InnerVolumeSpecName "kube-api-access-7tq7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:50.505965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.505943 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29c14fb-7305-431d-ad95-e51e4ea1b417-istiod-ca-cert" pod "c29c14fb-7305-431d-ad95-e51e4ea1b417" (UID: "c29c14fb-7305-431d-ad95-e51e4ea1b417"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:50.506427 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.506406 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-envoy" pod "c29c14fb-7305-431d-ad95-e51e4ea1b417" (UID: "c29c14fb-7305-431d-ad95-e51e4ea1b417"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.603586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.603479 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c14fb-7305-431d-ad95-e51e4ea1b417-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.603586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.603526 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rxkb\" (UniqueName: \"kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-kube-api-access-4rxkb\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.603586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.603542 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.603586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.603555 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.603586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.603568 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.603586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.603580 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.603586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.603593 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c14fb-7305-431d-ad95-e51e4ea1b417-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.603968 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.603608 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f544e1a-31c2-4601-a0d3-95f214fa213d-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.603968 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.603619 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tq7v\" (UniqueName: \"kubernetes.io/projected/c29c14fb-7305-431d-ad95-e51e4ea1b417-kube-api-access-7tq7v\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.769009 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.768971 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84"] Nov 02 20:29:50.787574 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:50.787542 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-qml84"] Nov 02 20:29:51.320423 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:51.320338 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f544e1a-31c2-4601-a0d3-95f214fa213d" path="/var/lib/kubelet/pods/8f544e1a-31c2-4601-a0d3-95f214fa213d/volumes" Nov 02 20:29:51.355386 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:51.355356 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" Nov 02 20:29:51.355555 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:51.355376 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9" event={"ID":"c29c14fb-7305-431d-ad95-e51e4ea1b417","Type":"ContainerDied","Data":"9639379c5339b967f15d211b405605bee8e5e1f79347d94bac5eb4f81eb78fbd"} Nov 02 20:29:51.355555 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:51.355446 5561 scope.go:122] "RemoveContainer" containerID="33d492eb5403cc48f9d844a19627396134dd6cad4f1f0b12e16db2b2f272e481" Nov 02 20:29:51.511547 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:51.511513 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9"] Nov 02 20:29:51.558994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:51.558958 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-8wzk9"] Nov 02 20:29:52.072603 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:52.072566 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:52.682540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:52.682505 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:52.875174 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:52.875135 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:53.232899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:53.228992 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:53.315196 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:53.315164 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29c14fb-7305-431d-ad95-e51e4ea1b417" path="/var/lib/kubelet/pods/c29c14fb-7305-431d-ad95-e51e4ea1b417/volumes" Nov 02 20:29:53.608718 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:53.608633 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:57.592497 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:57.592460 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:57.790542 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:57.790509 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:58.110337 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:29:58.110301 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:00.198953 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.198922 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd"] Nov 02 20:30:00.199641 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199623 5561 cpu_manager.go:414] "containerMap: removing container" podUID="8f544e1a-31c2-4601-a0d3-95f214fa213d" containerName="istio-proxy" Nov 02 20:30:00.199734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199642 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f544e1a-31c2-4601-a0d3-95f214fa213d" containerName="istio-proxy" Nov 02 20:30:00.199734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199659 5561 cpu_manager.go:414] "containerMap: removing container" podUID="c29c14fb-7305-431d-ad95-e51e4ea1b417" containerName="istio-proxy" Nov 02 20:30:00.199734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199664 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29c14fb-7305-431d-ad95-e51e4ea1b417" containerName="istio-proxy" Nov 02 20:30:00.199734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199691 5561 cpu_manager.go:414] "containerMap: removing container" podUID="ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a" containerName="payload-pre-processing" Nov 02 20:30:00.199734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199701 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a" containerName="payload-pre-processing" Nov 02 20:30:00.199734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199722 5561 cpu_manager.go:414] "containerMap: removing container" podUID="b6051c83-da4e-482c-8833-ca0dd2bc9bcf" containerName="payload-pre-processing" Nov 02 20:30:00.199734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199727 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6051c83-da4e-482c-8833-ca0dd2bc9bcf" containerName="payload-pre-processing" Nov 02 20:30:00.200041 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199753 5561 cpu_manager.go:414] "containerMap: removing container" podUID="6e8276a3-e24b-4a21-8522-c5bac20d5a3b" containerName="payload-processing" Nov 02 20:30:00.200041 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199761 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8276a3-e24b-4a21-8522-c5bac20d5a3b" containerName="payload-processing" Nov 02 20:30:00.200041 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199892 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6051c83-da4e-482c-8833-ca0dd2bc9bcf" containerName="payload-pre-processing" Nov 02 20:30:00.200041 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199909 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29c14fb-7305-431d-ad95-e51e4ea1b417" containerName="istio-proxy" Nov 02 20:30:00.200041 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199922 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8276a3-e24b-4a21-8522-c5bac20d5a3b" containerName="payload-processing" Nov 02 20:30:00.200041 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199935 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1e0f45-5cb6-4b98-b7db-4df3b3abcd0a" containerName="payload-pre-processing" Nov 02 20:30:00.200041 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.199944 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f544e1a-31c2-4601-a0d3-95f214fa213d" containerName="istio-proxy" Nov 02 20:30:00.203517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.203492 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" Nov 02 20:30:00.227904 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.227854 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk"] Nov 02 20:30:00.234160 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.234129 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" Nov 02 20:30:00.249058 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.249024 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvp46\" (UniqueName: \"kubernetes.io/projected/f7f03537-f830-4843-b04a-6c2de40b6234-kube-api-access-qvp46\") pod \"maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd\" (UID: \"f7f03537-f830-4843-b04a-6c2de40b6234\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" Nov 02 20:30:00.249214 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.249189 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f03537-f830-4843-b04a-6c2de40b6234-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd\" (UID: \"f7f03537-f830-4843-b04a-6c2de40b6234\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" Nov 02 20:30:00.249381 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.249359 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tslm4\" (UniqueName: \"kubernetes.io/projected/cca13fc0-df97-43b4-8238-38efc937ad55-kube-api-access-tslm4\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk\" (UID: \"cca13fc0-df97-43b4-8238-38efc937ad55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" Nov 02 20:30:00.249511 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.249489 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca13fc0-df97-43b4-8238-38efc937ad55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk\" (UID: \"cca13fc0-df97-43b4-8238-38efc937ad55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" Nov 02 20:30:00.324514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.324482 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:30:00.351271 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.351224 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tslm4\" (UniqueName: \"kubernetes.io/projected/cca13fc0-df97-43b4-8238-38efc937ad55-kube-api-access-tslm4\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk\" (UID: \"cca13fc0-df97-43b4-8238-38efc937ad55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" Nov 02 20:30:00.351454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.351345 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca13fc0-df97-43b4-8238-38efc937ad55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk\" (UID: \"cca13fc0-df97-43b4-8238-38efc937ad55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" Nov 02 20:30:00.351454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.351438 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvp46\" (UniqueName: \"kubernetes.io/projected/f7f03537-f830-4843-b04a-6c2de40b6234-kube-api-access-qvp46\") pod \"maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd\" (UID: \"f7f03537-f830-4843-b04a-6c2de40b6234\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" Nov 02 20:30:00.353163 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.353134 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca13fc0-df97-43b4-8238-38efc937ad55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk\" (UID: \"cca13fc0-df97-43b4-8238-38efc937ad55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" Nov 02 20:30:00.355236 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.354288 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f03537-f830-4843-b04a-6c2de40b6234-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd\" (UID: \"f7f03537-f830-4843-b04a-6c2de40b6234\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" Nov 02 20:30:00.355354 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.354925 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f03537-f830-4843-b04a-6c2de40b6234-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd\" (UID: \"f7f03537-f830-4843-b04a-6c2de40b6234\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" Nov 02 20:30:00.359509 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.359485 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk"] Nov 02 20:30:00.401804 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.401770 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd"] Nov 02 20:30:00.406454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.406417 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tslm4\" (UniqueName: \"kubernetes.io/projected/cca13fc0-df97-43b4-8238-38efc937ad55-kube-api-access-tslm4\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk\" (UID: \"cca13fc0-df97-43b4-8238-38efc937ad55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" Nov 02 20:30:00.408510 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.408484 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvp46\" (UniqueName: \"kubernetes.io/projected/f7f03537-f830-4843-b04a-6c2de40b6234-kube-api-access-qvp46\") pod \"maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd\" (UID: \"f7f03537-f830-4843-b04a-6c2de40b6234\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" Nov 02 20:30:00.522168 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.522089 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" Nov 02 20:30:00.553663 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.553636 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" Nov 02 20:30:00.896479 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.896386 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd"] Nov 02 20:30:00.902854 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:30:00.902827 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f03537_f830_4843_b04a_6c2de40b6234.slice/crio-a15c9151ee8b4125669a0b04d05e5f2344283e6f016f822d6fd9f3d28ed35436 WatchSource:0}: Error finding container a15c9151ee8b4125669a0b04d05e5f2344283e6f016f822d6fd9f3d28ed35436: Status 404 returned error can't find the container with id a15c9151ee8b4125669a0b04d05e5f2344283e6f016f822d6fd9f3d28ed35436 Nov 02 20:30:00.950592 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:00.950561 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk"] Nov 02 20:30:00.955754 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:30:00.955721 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca13fc0_df97_43b4_8238_38efc937ad55.slice/crio-0df398b0531607abeeba0df3472932fac69529b07077e2de2b350481b943ec69 WatchSource:0}: Error finding container 0df398b0531607abeeba0df3472932fac69529b07077e2de2b350481b943ec69: Status 404 returned error can't find the container with id 0df398b0531607abeeba0df3472932fac69529b07077e2de2b350481b943ec69 Nov 02 20:30:01.434020 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:01.433982 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" event={"ID":"f7f03537-f830-4843-b04a-6c2de40b6234","Type":"ContainerStarted","Data":"a15c9151ee8b4125669a0b04d05e5f2344283e6f016f822d6fd9f3d28ed35436"} Nov 02 20:30:01.435300 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:01.435272 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" event={"ID":"cca13fc0-df97-43b4-8238-38efc937ad55","Type":"ContainerStarted","Data":"0df398b0531607abeeba0df3472932fac69529b07077e2de2b350481b943ec69"} Nov 02 20:30:03.782089 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:03.782052 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:30:04.008792 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:04.008754 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:04.260605 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:04.260563 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:04.437004 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:04.436970 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:04.895754 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:04.895720 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:07.477784 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:07.477752 5561 generic.go:356] "Generic (PLEG): container finished" podID="f7f03537-f830-4843-b04a-6c2de40b6234" containerID="d0ed229ad74239a65399356f1c0cc04b3b1a6fd0c489a1d937c7399f81e0acb9" exitCode=60 Nov 02 20:30:07.478289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:07.477839 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" event={"ID":"f7f03537-f830-4843-b04a-6c2de40b6234","Type":"ContainerDied","Data":"d0ed229ad74239a65399356f1c0cc04b3b1a6fd0c489a1d937c7399f81e0acb9"} Nov 02 20:30:07.478289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:07.478272 5561 scope.go:122] "RemoveContainer" containerID="d0ed229ad74239a65399356f1c0cc04b3b1a6fd0c489a1d937c7399f81e0acb9" Nov 02 20:30:07.479646 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:07.479626 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" event={"ID":"cca13fc0-df97-43b4-8238-38efc937ad55","Type":"ContainerStarted","Data":"5dfc69f2440f06b63bc29bf088804544468aba5e7ff209301493bbfc5bd4837c"} Nov 02 20:30:07.691664 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:07.691546 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" podStartSLOduration=1.641795127 podStartE2EDuration="7.691531836s" totalImagesPullingTime="6.049736709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:07.691250689 +0000 UTC m=+2244.499034908" watchObservedRunningTime="2026-11-02 20:30:07.691531836 +0000 UTC m=+2244.499316049" Nov 02 20:30:08.487650 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:08.487571 5561 generic.go:356] "Generic (PLEG): container finished" podID="f7f03537-f830-4843-b04a-6c2de40b6234" containerID="796a15b8afccb7b0abc0fe5d69d65b2472ea4f891fd719f6cb049f85df73baf5" exitCode=60 Nov 02 20:30:08.488119 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:08.487657 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" event={"ID":"f7f03537-f830-4843-b04a-6c2de40b6234","Type":"ContainerDied","Data":"796a15b8afccb7b0abc0fe5d69d65b2472ea4f891fd719f6cb049f85df73baf5"} Nov 02 20:30:08.488119 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:08.487718 5561 scope.go:122] "RemoveContainer" containerID="d0ed229ad74239a65399356f1c0cc04b3b1a6fd0c489a1d937c7399f81e0acb9" Nov 02 20:30:08.488119 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:08.487935 5561 scope.go:122] "RemoveContainer" containerID="796a15b8afccb7b0abc0fe5d69d65b2472ea4f891fd719f6cb049f85df73baf5" Nov 02 20:30:08.488263 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:08.488238 5561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd_odh-ai-gateway-infra(f7f03537-f830-4843-b04a-6c2de40b6234)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" Nov 02 20:30:08.489362 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:08.489342 5561 generic.go:356] "Generic (PLEG): container finished" podID="cca13fc0-df97-43b4-8238-38efc937ad55" containerID="5dfc69f2440f06b63bc29bf088804544468aba5e7ff209301493bbfc5bd4837c" exitCode=60 Nov 02 20:30:08.489425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:08.489399 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" event={"ID":"cca13fc0-df97-43b4-8238-38efc937ad55","Type":"ContainerDied","Data":"5dfc69f2440f06b63bc29bf088804544468aba5e7ff209301493bbfc5bd4837c"} Nov 02 20:30:08.489628 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:08.489616 5561 scope.go:122] "RemoveContainer" containerID="5dfc69f2440f06b63bc29bf088804544468aba5e7ff209301493bbfc5bd4837c" Nov 02 20:30:09.000012 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:08.999862 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:09.498157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:09.498119 5561 scope.go:122] "RemoveContainer" containerID="796a15b8afccb7b0abc0fe5d69d65b2472ea4f891fd719f6cb049f85df73baf5" Nov 02 20:30:09.498614 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:09.498480 5561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd_odh-ai-gateway-infra(f7f03537-f830-4843-b04a-6c2de40b6234)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" Nov 02 20:30:09.499578 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:09.499554 5561 generic.go:356] "Generic (PLEG): container finished" podID="cca13fc0-df97-43b4-8238-38efc937ad55" containerID="130faa31c266bba74f17baea3a1c0ef8afef59ec3bee1b9766f675242f40fd25" exitCode=60 Nov 02 20:30:09.499706 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:09.499584 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" event={"ID":"cca13fc0-df97-43b4-8238-38efc937ad55","Type":"ContainerDied","Data":"130faa31c266bba74f17baea3a1c0ef8afef59ec3bee1b9766f675242f40fd25"} Nov 02 20:30:09.499706 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:09.499637 5561 scope.go:122] "RemoveContainer" containerID="5dfc69f2440f06b63bc29bf088804544468aba5e7ff209301493bbfc5bd4837c" Nov 02 20:30:09.499964 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:09.499947 5561 scope.go:122] "RemoveContainer" containerID="130faa31c266bba74f17baea3a1c0ef8afef59ec3bee1b9766f675242f40fd25" Nov 02 20:30:09.500277 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:09.500251 5561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk_odh-ai-gateway-infra(cca13fc0-df97-43b4-8238-38efc937ad55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" Nov 02 20:30:10.507782 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:10.507756 5561 scope.go:122] "RemoveContainer" containerID="130faa31c266bba74f17baea3a1c0ef8afef59ec3bee1b9766f675242f40fd25" Nov 02 20:30:10.508184 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:10.508095 5561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk_odh-ai-gateway-infra(cca13fc0-df97-43b4-8238-38efc937ad55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" Nov 02 20:30:10.728195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:10.728156 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:11.002286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:11.002247 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:11.192268 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:11.192231 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:13.572687 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:13.572645 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:15.056936 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:15.056902 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:15.246998 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:15.246963 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:18.320979 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:18.320942 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:18.525231 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:18.525182 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:18.722721 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:18.722690 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:21.306540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:21.306506 5561 scope.go:122] "RemoveContainer" containerID="796a15b8afccb7b0abc0fe5d69d65b2472ea4f891fd719f6cb049f85df73baf5" Nov 02 20:30:22.306289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:22.306260 5561 scope.go:122] "RemoveContainer" containerID="130faa31c266bba74f17baea3a1c0ef8afef59ec3bee1b9766f675242f40fd25" Nov 02 20:30:22.591086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:22.591056 5561 generic.go:356] "Generic (PLEG): container finished" podID="f7f03537-f830-4843-b04a-6c2de40b6234" containerID="c04a4a2b14dd343d1f18b160624b892d3c17c71f3dbb83d6d6c401379f2602d6" exitCode=60 Nov 02 20:30:22.591471 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:22.591133 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" event={"ID":"f7f03537-f830-4843-b04a-6c2de40b6234","Type":"ContainerDied","Data":"c04a4a2b14dd343d1f18b160624b892d3c17c71f3dbb83d6d6c401379f2602d6"} Nov 02 20:30:22.591471 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:22.591197 5561 scope.go:122] "RemoveContainer" containerID="796a15b8afccb7b0abc0fe5d69d65b2472ea4f891fd719f6cb049f85df73baf5" Nov 02 20:30:22.591595 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:22.591523 5561 scope.go:122] "RemoveContainer" containerID="c04a4a2b14dd343d1f18b160624b892d3c17c71f3dbb83d6d6c401379f2602d6" Nov 02 20:30:22.591865 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:22.591845 5561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd_odh-ai-gateway-infra(f7f03537-f830-4843-b04a-6c2de40b6234)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" Nov 02 20:30:23.147728 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:23.147697 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:23.408284 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:23.408242 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:23.615940 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:23.615866 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:23.620591 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:23.620557 5561 generic.go:356] "Generic (PLEG): container finished" podID="cca13fc0-df97-43b4-8238-38efc937ad55" containerID="b21d3fc963dfdf37cfe2498271a91e21c1e617641012b28f3b9900c82b7c4713" exitCode=60 Nov 02 20:30:23.621096 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:23.620811 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" event={"ID":"cca13fc0-df97-43b4-8238-38efc937ad55","Type":"ContainerDied","Data":"b21d3fc963dfdf37cfe2498271a91e21c1e617641012b28f3b9900c82b7c4713"} Nov 02 20:30:23.621311 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:23.621297 5561 scope.go:122] "RemoveContainer" containerID="130faa31c266bba74f17baea3a1c0ef8afef59ec3bee1b9766f675242f40fd25" Nov 02 20:30:23.622228 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:23.622202 5561 scope.go:122] "RemoveContainer" containerID="b21d3fc963dfdf37cfe2498271a91e21c1e617641012b28f3b9900c82b7c4713" Nov 02 20:30:23.624844 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:23.624663 5561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk_odh-ai-gateway-infra(cca13fc0-df97-43b4-8238-38efc937ad55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" Nov 02 20:30:23.722693 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:23.722554 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd"] Nov 02 20:30:23.888505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:23.888472 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" Nov 02 20:30:23.995951 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:23.995557 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f03537-f830-4843-b04a-6c2de40b6234-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f03537-f830-4843-b04a-6c2de40b6234-maas-api-service-ca\") pod \"f7f03537-f830-4843-b04a-6c2de40b6234\" (UID: \"f7f03537-f830-4843-b04a-6c2de40b6234\") " Nov 02 20:30:23.995951 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:23.995701 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f03537-f830-4843-b04a-6c2de40b6234-kube-api-access-qvp46\" (UniqueName: \"kubernetes.io/projected/f7f03537-f830-4843-b04a-6c2de40b6234-kube-api-access-qvp46\") pod \"f7f03537-f830-4843-b04a-6c2de40b6234\" (UID: \"f7f03537-f830-4843-b04a-6c2de40b6234\") " Nov 02 20:30:24.000303 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:23.999701 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f03537-f830-4843-b04a-6c2de40b6234-maas-api-service-ca" pod "f7f03537-f830-4843-b04a-6c2de40b6234" (UID: "f7f03537-f830-4843-b04a-6c2de40b6234"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:24.015380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:24.011051 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f03537-f830-4843-b04a-6c2de40b6234-kube-api-access-qvp46" pod "f7f03537-f830-4843-b04a-6c2de40b6234" (UID: "f7f03537-f830-4843-b04a-6c2de40b6234"). InnerVolumeSpecName "kube-api-access-qvp46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:24.097344 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:24.097311 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f03537-f830-4843-b04a-6c2de40b6234-maas-api-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.097344 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:24.097342 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvp46\" (UniqueName: \"kubernetes.io/projected/f7f03537-f830-4843-b04a-6c2de40b6234-kube-api-access-qvp46\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.216380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:24.216342 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:24.628787 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:24.628751 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" Nov 02 20:30:24.628787 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:24.628772 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd" event={"ID":"f7f03537-f830-4843-b04a-6c2de40b6234","Type":"ContainerDied","Data":"a15c9151ee8b4125669a0b04d05e5f2344283e6f016f822d6fd9f3d28ed35436"} Nov 02 20:30:24.629384 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:24.628849 5561 scope.go:122] "RemoveContainer" containerID="c04a4a2b14dd343d1f18b160624b892d3c17c71f3dbb83d6d6c401379f2602d6" Nov 02 20:30:24.754342 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:24.754310 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk"] Nov 02 20:30:24.789939 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:24.789897 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd"] Nov 02 20:30:24.793113 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:24.793082 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-accdf5-29894190-lpncd"] Nov 02 20:30:24.927312 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:24.927286 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" Nov 02 20:30:25.009225 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.009188 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca13fc0-df97-43b4-8238-38efc937ad55-kube-api-access-tslm4\" (UniqueName: \"kubernetes.io/projected/cca13fc0-df97-43b4-8238-38efc937ad55-kube-api-access-tslm4\") pod \"cca13fc0-df97-43b4-8238-38efc937ad55\" (UID: \"cca13fc0-df97-43b4-8238-38efc937ad55\") " Nov 02 20:30:25.009484 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.009464 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca13fc0-df97-43b4-8238-38efc937ad55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca13fc0-df97-43b4-8238-38efc937ad55-maas-api-service-ca\") pod \"cca13fc0-df97-43b4-8238-38efc937ad55\" (UID: \"cca13fc0-df97-43b4-8238-38efc937ad55\") " Nov 02 20:30:25.009933 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.009899 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca13fc0-df97-43b4-8238-38efc937ad55-maas-api-service-ca" pod "cca13fc0-df97-43b4-8238-38efc937ad55" (UID: "cca13fc0-df97-43b4-8238-38efc937ad55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:25.010071 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.010031 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca13fc0-df97-43b4-8238-38efc937ad55-maas-api-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:25.011435 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.011398 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca13fc0-df97-43b4-8238-38efc937ad55-kube-api-access-tslm4" pod "cca13fc0-df97-43b4-8238-38efc937ad55" (UID: "cca13fc0-df97-43b4-8238-38efc937ad55"). InnerVolumeSpecName "kube-api-access-tslm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:25.111211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.111180 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tslm4\" (UniqueName: \"kubernetes.io/projected/cca13fc0-df97-43b4-8238-38efc937ad55-kube-api-access-tslm4\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:25.314791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.314759 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" path="/var/lib/kubelet/pods/f7f03537-f830-4843-b04a-6c2de40b6234/volumes" Nov 02 20:30:25.641921 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.641809 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" event={"ID":"cca13fc0-df97-43b4-8238-38efc937ad55","Type":"ContainerDied","Data":"0df398b0531607abeeba0df3472932fac69529b07077e2de2b350481b943ec69"} Nov 02 20:30:25.641921 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.641895 5561 scope.go:122] "RemoveContainer" containerID="b21d3fc963dfdf37cfe2498271a91e21c1e617641012b28f3b9900c82b7c4713" Nov 02 20:30:25.641921 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.641825 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk" Nov 02 20:30:25.798241 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.798205 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk"] Nov 02 20:30:25.845932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.845887 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5fc50d-29894190-lm8zk"] Nov 02 20:30:25.854754 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:25.854725 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:26.031517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:26.031484 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:27.320217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:27.320175 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" path="/var/lib/kubelet/pods/cca13fc0-df97-43b4-8238-38efc937ad55/volumes" Nov 02 20:30:28.027869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:28.027838 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:28.210820 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:28.210786 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:28.395062 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:28.394968 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:29.492220 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:29.492181 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:29.675166 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:29.675134 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:31.271305 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:31.271270 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:31.455200 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:31.454727 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:32.477126 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.477091 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk"] Nov 02 20:30:32.477566 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.477310 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" podUID="daa3a147-4a02-49a7-8179-2045d8fd9489" containerName="maas-api" containerID="cri-o://69eb7ce38c5ad4ec4baccb3c011943dc22e6a740b2e9e11d134a2084012b6fc8" gracePeriod=30 Nov 02 20:30:32.621743 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.621697 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:32.756177 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.756037 5561 generic.go:356] "Generic (PLEG): container finished" podID="daa3a147-4a02-49a7-8179-2045d8fd9489" containerID="69eb7ce38c5ad4ec4baccb3c011943dc22e6a740b2e9e11d134a2084012b6fc8" exitCode=0 Nov 02 20:30:32.756646 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.756403 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" event={"ID":"daa3a147-4a02-49a7-8179-2045d8fd9489","Type":"ContainerDied","Data":"69eb7ce38c5ad4ec4baccb3c011943dc22e6a740b2e9e11d134a2084012b6fc8"} Nov 02 20:30:32.826421 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.826392 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:30:32.907898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.905790 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa3a147-4a02-49a7-8179-2045d8fd9489-kube-api-access-rn2hw\" (UniqueName: \"kubernetes.io/projected/daa3a147-4a02-49a7-8179-2045d8fd9489-kube-api-access-rn2hw\") pod \"daa3a147-4a02-49a7-8179-2045d8fd9489\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " Nov 02 20:30:32.907898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.905912 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-maas-api-tls\") pod \"daa3a147-4a02-49a7-8179-2045d8fd9489\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " Nov 02 20:30:32.907898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.906042 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa3a147-4a02-49a7-8179-2045d8fd9489-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa3a147-4a02-49a7-8179-2045d8fd9489-openshift-service-ca\") pod \"daa3a147-4a02-49a7-8179-2045d8fd9489\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " Nov 02 20:30:32.907898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.906141 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-metrics-certs\") pod \"daa3a147-4a02-49a7-8179-2045d8fd9489\" (UID: \"daa3a147-4a02-49a7-8179-2045d8fd9489\") " Nov 02 20:30:32.908901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.908289 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa3a147-4a02-49a7-8179-2045d8fd9489-openshift-service-ca" pod "daa3a147-4a02-49a7-8179-2045d8fd9489" (UID: "daa3a147-4a02-49a7-8179-2045d8fd9489"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:32.913381 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.913335 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-metrics-certs" pod "daa3a147-4a02-49a7-8179-2045d8fd9489" (UID: "daa3a147-4a02-49a7-8179-2045d8fd9489"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:32.923466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.923411 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa3a147-4a02-49a7-8179-2045d8fd9489-kube-api-access-rn2hw" pod "daa3a147-4a02-49a7-8179-2045d8fd9489" (UID: "daa3a147-4a02-49a7-8179-2045d8fd9489"). InnerVolumeSpecName "kube-api-access-rn2hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:32.923709 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:32.923679 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-maas-api-tls" pod "daa3a147-4a02-49a7-8179-2045d8fd9489" (UID: "daa3a147-4a02-49a7-8179-2045d8fd9489"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:33.007770 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.007662 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn2hw\" (UniqueName: \"kubernetes.io/projected/daa3a147-4a02-49a7-8179-2045d8fd9489-kube-api-access-rn2hw\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:33.007770 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.007702 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:33.007770 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.007717 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa3a147-4a02-49a7-8179-2045d8fd9489-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:33.007770 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.007731 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa3a147-4a02-49a7-8179-2045d8fd9489-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:33.201240 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.201203 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr"] Nov 02 20:30:33.201429 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.201408 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" podUID="0284d6e4-70b6-467c-a101-1872f264a098" containerName="discovery" containerID="cri-o://e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837" gracePeriod=30 Nov 02 20:30:33.618305 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.618274 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:30:33.633458 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.633423 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-cacerts\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-cacerts\") pod \"0284d6e4-70b6-467c-a101-1872f264a098\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " Nov 02 20:30:33.633624 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.633492 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-ca-configmap\") pod \"0284d6e4-70b6-467c-a101-1872f264a098\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " Nov 02 20:30:33.633686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.633620 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-kube-api-access-m628t\" (UniqueName: \"kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-kube-api-access-m628t\") pod \"0284d6e4-70b6-467c-a101-1872f264a098\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " Nov 02 20:30:33.633686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.633678 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-dns-cert\") pod \"0284d6e4-70b6-467c-a101-1872f264a098\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " Nov 02 20:30:33.633907 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.633861 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-kubeconfig\") pod \"0284d6e4-70b6-467c-a101-1872f264a098\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " Nov 02 20:30:33.633979 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.633950 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0284d6e4-70b6-467c-a101-1872f264a098-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0284d6e4-70b6-467c-a101-1872f264a098-local-certs\") pod \"0284d6e4-70b6-467c-a101-1872f264a098\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " Nov 02 20:30:33.634034 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.633859 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-ca-configmap" pod "0284d6e4-70b6-467c-a101-1872f264a098" (UID: "0284d6e4-70b6-467c-a101-1872f264a098"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:33.634034 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.634025 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-istio-token\" (UniqueName: \"kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-istio-token\") pod \"0284d6e4-70b6-467c-a101-1872f264a098\" (UID: \"0284d6e4-70b6-467c-a101-1872f264a098\") " Nov 02 20:30:33.634752 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.634725 5561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-ca-configmap\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:33.636543 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.636515 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-kube-api-access-m628t" pod "0284d6e4-70b6-467c-a101-1872f264a098" (UID: "0284d6e4-70b6-467c-a101-1872f264a098"). InnerVolumeSpecName "kube-api-access-m628t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:33.636913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.636765 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-cacerts" pod "0284d6e4-70b6-467c-a101-1872f264a098" (UID: "0284d6e4-70b6-467c-a101-1872f264a098"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:33.636913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.636810 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-dns-cert" pod "0284d6e4-70b6-467c-a101-1872f264a098" (UID: "0284d6e4-70b6-467c-a101-1872f264a098"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:33.636913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.636860 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0284d6e4-70b6-467c-a101-1872f264a098-local-certs" pod "0284d6e4-70b6-467c-a101-1872f264a098" (UID: "0284d6e4-70b6-467c-a101-1872f264a098"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:33.637151 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.637126 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-kubeconfig" pod "0284d6e4-70b6-467c-a101-1872f264a098" (UID: "0284d6e4-70b6-467c-a101-1872f264a098"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:33.638100 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.638073 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-istio-token" pod "0284d6e4-70b6-467c-a101-1872f264a098" (UID: "0284d6e4-70b6-467c-a101-1872f264a098"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:33.736398 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.736360 5561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-cacerts\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:33.736555 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.736402 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m628t\" (UniqueName: \"kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-kube-api-access-m628t\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:33.736555 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.736418 5561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-csr-dns-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:33.736555 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.736432 5561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0284d6e4-70b6-467c-a101-1872f264a098-istio-kubeconfig\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:33.736555 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.736444 5561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0284d6e4-70b6-467c-a101-1872f264a098-local-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:33.736555 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.736455 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0284d6e4-70b6-467c-a101-1872f264a098-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:33.765420 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.765386 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" Nov 02 20:30:33.765602 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.765416 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk" event={"ID":"daa3a147-4a02-49a7-8179-2045d8fd9489","Type":"ContainerDied","Data":"abaa525b4f49c967760e372a114f485cf0763f4776b3df6084f7816f7ffacae4"} Nov 02 20:30:33.765602 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.765479 5561 scope.go:122] "RemoveContainer" containerID="69eb7ce38c5ad4ec4baccb3c011943dc22e6a740b2e9e11d134a2084012b6fc8" Nov 02 20:30:33.767250 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.767217 5561 generic.go:356] "Generic (PLEG): container finished" podID="0284d6e4-70b6-467c-a101-1872f264a098" containerID="e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837" exitCode=0 Nov 02 20:30:33.767348 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.767267 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" Nov 02 20:30:33.767416 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.767393 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" event={"ID":"0284d6e4-70b6-467c-a101-1872f264a098","Type":"ContainerDied","Data":"e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837"} Nov 02 20:30:33.768362 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.767441 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr" event={"ID":"0284d6e4-70b6-467c-a101-1872f264a098","Type":"ContainerDied","Data":"86896826a2798c9d5600e1bc48facb58dcffbc27a0a53ed503dfbd20ef08a945"} Nov 02 20:30:33.781236 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.780485 5561 scope.go:122] "RemoveContainer" containerID="e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837" Nov 02 20:30:33.788757 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:33.788720 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:30:33.792040 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.792020 5561 scope.go:122] "RemoveContainer" containerID="e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837" Nov 02 20:30:33.792349 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:33.792327 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837\": container with ID starting with e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837 not found: ID does not exist" containerID="e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837" Nov 02 20:30:33.792400 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.792356 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837"} err="failed to get container status \"e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837\": rpc error: code = NotFound desc = could not find container \"e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837\": container with ID starting with e2ec5bcd929167b94e23790813ed63842f3d876602f3c91a5f9e8553a822a837 not found: ID does not exist" Nov 02 20:30:33.922623 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.922587 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk"] Nov 02 20:30:33.926131 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:33.926102 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-mz6fk"] Nov 02 20:30:34.031207 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:34.031177 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr"] Nov 02 20:30:34.034323 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:34.034299 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4lhbr"] Nov 02 20:30:34.196618 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:34.196533 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:34.464859 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:34.464769 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:34.635728 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:34.635689 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:35.091112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:35.091080 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:35.314177 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:35.314148 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0284d6e4-70b6-467c-a101-1872f264a098" path="/var/lib/kubelet/pods/0284d6e4-70b6-467c-a101-1872f264a098/volumes" Nov 02 20:30:35.314596 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:35.314583 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa3a147-4a02-49a7-8179-2045d8fd9489" path="/var/lib/kubelet/pods/daa3a147-4a02-49a7-8179-2045d8fd9489/volumes" Nov 02 20:30:39.077853 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.077816 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8"] Nov 02 20:30:39.078400 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078385 5561 cpu_manager.go:414] "containerMap: removing container" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" containerName="cleanup" Nov 02 20:30:39.078449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078402 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" containerName="cleanup" Nov 02 20:30:39.078449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078414 5561 cpu_manager.go:414] "containerMap: removing container" podUID="daa3a147-4a02-49a7-8179-2045d8fd9489" containerName="maas-api" Nov 02 20:30:39.078449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078419 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa3a147-4a02-49a7-8179-2045d8fd9489" containerName="maas-api" Nov 02 20:30:39.078449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078437 5561 cpu_manager.go:414] "containerMap: removing container" podUID="0284d6e4-70b6-467c-a101-1872f264a098" containerName="discovery" Nov 02 20:30:39.078449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078442 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0284d6e4-70b6-467c-a101-1872f264a098" containerName="discovery" Nov 02 20:30:39.078598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078452 5561 cpu_manager.go:414] "containerMap: removing container" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" containerName="cleanup" Nov 02 20:30:39.078598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078460 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" containerName="cleanup" Nov 02 20:30:39.078598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078475 5561 cpu_manager.go:414] "containerMap: removing container" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" containerName="cleanup" Nov 02 20:30:39.078598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078479 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" containerName="cleanup" Nov 02 20:30:39.078598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078493 5561 cpu_manager.go:414] "containerMap: removing container" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" containerName="cleanup" Nov 02 20:30:39.078598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078497 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" containerName="cleanup" Nov 02 20:30:39.078598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078511 5561 cpu_manager.go:414] "containerMap: removing container" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" containerName="cleanup" Nov 02 20:30:39.078598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078516 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" containerName="cleanup" Nov 02 20:30:39.079094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078603 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" containerName="cleanup" Nov 02 20:30:39.079094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078611 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa3a147-4a02-49a7-8179-2045d8fd9489" containerName="maas-api" Nov 02 20:30:39.079094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078620 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="0284d6e4-70b6-467c-a101-1872f264a098" containerName="discovery" Nov 02 20:30:39.079094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078625 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" containerName="cleanup" Nov 02 20:30:39.079094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078632 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" containerName="cleanup" Nov 02 20:30:39.079094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078641 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" containerName="cleanup" Nov 02 20:30:39.079094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.078645 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" containerName="cleanup" Nov 02 20:30:39.082125 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.082106 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.109597 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.109555 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/657eca17-2790-4b44-9264-40048be97ab1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.109765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.109607 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbtvg\" (UniqueName: \"kubernetes.io/projected/657eca17-2790-4b44-9264-40048be97ab1-kube-api-access-kbtvg\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.109813 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.109782 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-metrics-certs\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.109994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.109976 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.132033 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.132002 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1ab501-serving-cert\"" Nov 02 20:30:39.182168 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.182136 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8"] Nov 02 20:30:39.211362 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.211329 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-metrics-certs\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.211543 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.211494 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.211609 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.211560 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/657eca17-2790-4b44-9264-40048be97ab1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.211609 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.211589 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtvg\" (UniqueName: \"kubernetes.io/projected/657eca17-2790-4b44-9264-40048be97ab1-kube-api-access-kbtvg\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.211715 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:39.211650 5561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-serving-cert: secret "maas-api-e2e-inf-a-1ab501-serving-cert" not found Nov 02 20:30:39.211772 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:39.211742 5561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls podName:657eca17-2790-4b44-9264-40048be97ab1 nodeName:}" failed. No retries permitted until 2026-11-02 20:30:39.711719695 +0000 UTC m=+2276.519503909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls") pod "maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" (UID: "657eca17-2790-4b44-9264-40048be97ab1") : secret "maas-api-e2e-inf-a-1ab501-serving-cert" not found Nov 02 20:30:39.212217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.212196 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/657eca17-2790-4b44-9264-40048be97ab1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.214300 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.214274 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-metrics-certs\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.268184 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.268140 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbtvg\" (UniqueName: \"kubernetes.io/projected/657eca17-2790-4b44-9264-40048be97ab1-kube-api-access-kbtvg\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.342795 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.342711 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:39.538475 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.538435 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:39.544457 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.544420 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt"] Nov 02 20:30:39.544708 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.544674 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" containerID="cri-o://d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866" gracePeriod=30 Nov 02 20:30:39.718733 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:39.718697 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:39.718957 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:39.718856 5561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-serving-cert: secret "maas-api-e2e-inf-a-1ab501-serving-cert" not found Nov 02 20:30:39.718957 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:39.718946 5561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls podName:657eca17-2790-4b44-9264-40048be97ab1 nodeName:}" failed. No retries permitted until 2026-11-02 20:30:40.718930979 +0000 UTC m=+2277.526715173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls") pod "maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" (UID: "657eca17-2790-4b44-9264-40048be97ab1") : secret "maas-api-e2e-inf-a-1ab501-serving-cert" not found Nov 02 20:30:40.023331 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:40.023246 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:40.213704 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:40.213663 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:40.224966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:40.224932 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb"] Nov 02 20:30:40.225247 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:40.225169 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" containerID="cri-o://ab333609cbdce4df06882eb4a69986f06a9c6b2b48b016aae0218aa554f7cac1" gracePeriod=30 Nov 02 20:30:40.232920 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:40.228456 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx"] Nov 02 20:30:40.232920 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:40.228684 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" podUID="0fb1be7a-a4de-40ee-9805-dac41de74c30" containerName="istio-proxy" containerID="cri-o://55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103" gracePeriod=30 Nov 02 20:30:40.232920 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:40.232299 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc"] Nov 02 20:30:40.232920 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:40.232461 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" containerID="cri-o://1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438" gracePeriod=30 Nov 02 20:30:40.733987 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:40.733952 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:40.736359 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:40.736336 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls\") pod \"maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:40.895349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:40.895313 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:41.273233 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:41.273196 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8"] Nov 02 20:30:41.281292 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:30:41.281266 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657eca17_2790_4b44_9264_40048be97ab1.slice/crio-407091213bbc444effee7babfc93d521e7543468510fc21d1958ab2e698f7dfb WatchSource:0}: Error finding container 407091213bbc444effee7babfc93d521e7543468510fc21d1958ab2e698f7dfb: Status 404 returned error can't find the container with id 407091213bbc444effee7babfc93d521e7543468510fc21d1958ab2e698f7dfb Nov 02 20:30:41.820770 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:41.820735 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" event={"ID":"657eca17-2790-4b44-9264-40048be97ab1","Type":"ContainerStarted","Data":"407091213bbc444effee7babfc93d521e7543468510fc21d1958ab2e698f7dfb"} Nov 02 20:30:42.828705 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:42.828668 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" event={"ID":"657eca17-2790-4b44-9264-40048be97ab1","Type":"ContainerStarted","Data":"d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b"} Nov 02 20:30:42.829149 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:42.828868 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:42.939411 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:42.939354 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" podStartSLOduration=2.995913967 podStartE2EDuration="3.939339084s" totalImagesPullingTime="943.425117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:42.937441559 +0000 UTC m=+2279.745225776" watchObservedRunningTime="2026-11-02 20:30:42.939339084 +0000 UTC m=+2279.747123300" Nov 02 20:30:44.446605 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.446557 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:44.662484 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.662454 5561 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 20:30:44.662613 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.662518 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 20:30:44.806768 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.806743 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:30:44.842561 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.842525 5561 generic.go:356] "Generic (PLEG): container finished" podID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerID="d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866" exitCode=0 Nov 02 20:30:44.842712 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.842577 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" Nov 02 20:30:44.842712 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.842591 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" event={"ID":"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66","Type":"ContainerDied","Data":"d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866"} Nov 02 20:30:44.842712 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.842672 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt" event={"ID":"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66","Type":"ContainerDied","Data":"75cf158d1461d1e0be3d33ae00ea5ec6b56341dab5e4e98aa4ecfbc419455a19"} Nov 02 20:30:44.842712 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.842702 5561 scope.go:122] "RemoveContainer" containerID="d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866" Nov 02 20:30:44.852213 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.852192 5561 scope.go:122] "RemoveContainer" containerID="d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866" Nov 02 20:30:44.852518 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:44.852498 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866\": container with ID starting with d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866 not found: ID does not exist" containerID="d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866" Nov 02 20:30:44.852570 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.852526 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866"} err="failed to get container status \"d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866\": rpc error: code = NotFound desc = could not find container \"d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866\": container with ID starting with d8e386f2c015930c9556e661df6e66d200949309f6b35d7971a1cb540fe60866 not found: ID does not exist" Nov 02 20:30:44.889817 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.889777 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-socket\") pod \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " Nov 02 20:30:44.890035 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.889831 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-podinfo\") pod \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " Nov 02 20:30:44.890035 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.889866 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-certs\") pod \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " Nov 02 20:30:44.890035 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.889947 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-credential-socket\") pod \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " Nov 02 20:30:44.890035 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.889990 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-envoy\") pod \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " Nov 02 20:30:44.890274 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.890072 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-token\" (UniqueName: \"kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-token\") pod \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " Nov 02 20:30:44.890274 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.890097 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-socket" pod "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" (UID: "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:44.890274 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.890159 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-kube-api-access-wrz82\" (UniqueName: \"kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-kube-api-access-wrz82\") pod \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " Nov 02 20:30:44.890274 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.890247 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-data\") pod \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " Nov 02 20:30:44.890478 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.890291 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-certs" pod "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" (UID: "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:44.890478 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.890342 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-credential-socket" pod "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" (UID: "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:44.890478 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.890356 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istiod-ca-cert\") pod \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\" (UID: \"ef1780ec-55b0-41e8-bd7d-ba8f7d369f66\") " Nov 02 20:30:44.890612 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.890492 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-data" pod "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" (UID: "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:44.890712 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.890694 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istiod-ca-cert" pod "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" (UID: "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:44.891005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.890982 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:44.891112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.891007 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:44.891112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.891022 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:44.891112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.891035 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:44.891112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.891052 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:44.892370 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.892338 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-envoy" pod "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" (UID: "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:44.892485 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.892375 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-podinfo" pod "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" (UID: "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:30:44.892545 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.892527 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-kube-api-access-wrz82" pod "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" (UID: "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66"). InnerVolumeSpecName "kube-api-access-wrz82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:44.892598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.892555 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-token" pod "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" (UID: "ef1780ec-55b0-41e8-bd7d-ba8f7d369f66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:44.927941 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.927909 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-q5tqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:30:44.928087 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.927980 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:30:44.993299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.993206 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:44.993299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.993251 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrz82\" (UniqueName: \"kubernetes.io/projected/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-kube-api-access-wrz82\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:44.993299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.993270 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:44.993299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:44.993283 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.258381 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.258351 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt"] Nov 02 20:30:45.263131 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.263103 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-vhxpt"] Nov 02 20:30:45.316183 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.316152 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" path="/var/lib/kubelet/pods/ef1780ec-55b0-41e8-bd7d-ba8f7d369f66/volumes" Nov 02 20:30:45.496126 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.496063 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:30:45.560392 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.560367 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:30:45.601104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.601073 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-envoy\") pod \"9814eaca-0ca2-4925-ae97-ea136e05daff\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " Nov 02 20:30:45.601229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.601172 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-token\" (UniqueName: \"kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-token\") pod \"9814eaca-0ca2-4925-ae97-ea136e05daff\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " Nov 02 20:30:45.601286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.601247 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-data\") pod \"9814eaca-0ca2-4925-ae97-ea136e05daff\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " Nov 02 20:30:45.601358 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.601326 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9814eaca-0ca2-4925-ae97-ea136e05daff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9814eaca-0ca2-4925-ae97-ea136e05daff-istiod-ca-cert\") pod \"9814eaca-0ca2-4925-ae97-ea136e05daff\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " Nov 02 20:30:45.601418 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.601405 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-certs\") pod \"9814eaca-0ca2-4925-ae97-ea136e05daff\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " Nov 02 20:30:45.601485 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.601458 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-kube-api-access-fffxs\" (UniqueName: \"kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-kube-api-access-fffxs\") pod \"9814eaca-0ca2-4925-ae97-ea136e05daff\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " Nov 02 20:30:45.601543 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.601508 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-credential-socket\") pod \"9814eaca-0ca2-4925-ae97-ea136e05daff\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " Nov 02 20:30:45.601602 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.601564 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-socket\") pod \"9814eaca-0ca2-4925-ae97-ea136e05daff\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " Nov 02 20:30:45.601735 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.601696 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-data" pod "9814eaca-0ca2-4925-ae97-ea136e05daff" (UID: "9814eaca-0ca2-4925-ae97-ea136e05daff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.601815 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.601731 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-podinfo\") pod \"9814eaca-0ca2-4925-ae97-ea136e05daff\" (UID: \"9814eaca-0ca2-4925-ae97-ea136e05daff\") " Nov 02 20:30:45.602477 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.602273 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-credential-socket" pod "9814eaca-0ca2-4925-ae97-ea136e05daff" (UID: "9814eaca-0ca2-4925-ae97-ea136e05daff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.602477 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.602318 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.602477 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.602436 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-socket" pod "9814eaca-0ca2-4925-ae97-ea136e05daff" (UID: "9814eaca-0ca2-4925-ae97-ea136e05daff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.602791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.602581 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9814eaca-0ca2-4925-ae97-ea136e05daff-istiod-ca-cert" pod "9814eaca-0ca2-4925-ae97-ea136e05daff" (UID: "9814eaca-0ca2-4925-ae97-ea136e05daff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:45.603016 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.602994 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-certs" pod "9814eaca-0ca2-4925-ae97-ea136e05daff" (UID: "9814eaca-0ca2-4925-ae97-ea136e05daff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.604026 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.603999 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-envoy" pod "9814eaca-0ca2-4925-ae97-ea136e05daff" (UID: "9814eaca-0ca2-4925-ae97-ea136e05daff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.604622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.604595 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-podinfo" pod "9814eaca-0ca2-4925-ae97-ea136e05daff" (UID: "9814eaca-0ca2-4925-ae97-ea136e05daff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:30:45.605341 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.605315 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-kube-api-access-fffxs" pod "9814eaca-0ca2-4925-ae97-ea136e05daff" (UID: "9814eaca-0ca2-4925-ae97-ea136e05daff"). InnerVolumeSpecName "kube-api-access-fffxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:45.605407 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.605354 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-token" pod "9814eaca-0ca2-4925-ae97-ea136e05daff" (UID: "9814eaca-0ca2-4925-ae97-ea136e05daff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:45.703732 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.703695 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-token\" (UniqueName: \"kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-token\") pod \"0fb1be7a-a4de-40ee-9805-dac41de74c30\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " Nov 02 20:30:45.703919 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.703742 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb1be7a-a4de-40ee-9805-dac41de74c30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fb1be7a-a4de-40ee-9805-dac41de74c30-istiod-ca-cert\") pod \"0fb1be7a-a4de-40ee-9805-dac41de74c30\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " Nov 02 20:30:45.703919 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.703912 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-credential-socket\") pod \"0fb1be7a-a4de-40ee-9805-dac41de74c30\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " Nov 02 20:30:45.704062 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.703956 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-socket\") pod \"0fb1be7a-a4de-40ee-9805-dac41de74c30\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " Nov 02 20:30:45.704062 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.703992 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-certs\") pod \"0fb1be7a-a4de-40ee-9805-dac41de74c30\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " Nov 02 20:30:45.704062 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.704034 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-podinfo\") pod \"0fb1be7a-a4de-40ee-9805-dac41de74c30\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " Nov 02 20:30:45.704217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.704072 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-kube-api-access-jwqg2\" (UniqueName: \"kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-kube-api-access-jwqg2\") pod \"0fb1be7a-a4de-40ee-9805-dac41de74c30\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " Nov 02 20:30:45.704217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.704131 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-envoy\") pod \"0fb1be7a-a4de-40ee-9805-dac41de74c30\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " Nov 02 20:30:45.704318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.704228 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-data\") pod \"0fb1be7a-a4de-40ee-9805-dac41de74c30\" (UID: \"0fb1be7a-a4de-40ee-9805-dac41de74c30\") " Nov 02 20:30:45.704318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.704256 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-socket" pod "0fb1be7a-a4de-40ee-9805-dac41de74c30" (UID: "0fb1be7a-a4de-40ee-9805-dac41de74c30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.704318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.704270 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-certs" pod "0fb1be7a-a4de-40ee-9805-dac41de74c30" (UID: "0fb1be7a-a4de-40ee-9805-dac41de74c30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.704318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.704298 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb1be7a-a4de-40ee-9805-dac41de74c30-istiod-ca-cert" pod "0fb1be7a-a4de-40ee-9805-dac41de74c30" (UID: "0fb1be7a-a4de-40ee-9805-dac41de74c30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:45.704318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.704306 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-credential-socket" pod "0fb1be7a-a4de-40ee-9805-dac41de74c30" (UID: "0fb1be7a-a4de-40ee-9805-dac41de74c30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.704571 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.704554 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-data" pod "0fb1be7a-a4de-40ee-9805-dac41de74c30" (UID: "0fb1be7a-a4de-40ee-9805-dac41de74c30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.705187 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705159 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.705299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705190 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.705299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705205 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.705299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705218 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.705299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705232 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.705299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705247 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.705299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705260 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9814eaca-0ca2-4925-ae97-ea136e05daff-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.705299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705275 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.705299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705288 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.705739 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705302 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fffxs\" (UniqueName: \"kubernetes.io/projected/9814eaca-0ca2-4925-ae97-ea136e05daff-kube-api-access-fffxs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.705739 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705317 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.705739 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705330 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9814eaca-0ca2-4925-ae97-ea136e05daff-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.705739 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.705343 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fb1be7a-a4de-40ee-9805-dac41de74c30-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.706330 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.706303 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-podinfo" pod "0fb1be7a-a4de-40ee-9805-dac41de74c30" (UID: "0fb1be7a-a4de-40ee-9805-dac41de74c30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:30:45.706614 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.706590 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-envoy" pod "0fb1be7a-a4de-40ee-9805-dac41de74c30" (UID: "0fb1be7a-a4de-40ee-9805-dac41de74c30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.706865 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.706828 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-kube-api-access-jwqg2" pod "0fb1be7a-a4de-40ee-9805-dac41de74c30" (UID: "0fb1be7a-a4de-40ee-9805-dac41de74c30"). InnerVolumeSpecName "kube-api-access-jwqg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:45.706865 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.706858 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-token" pod "0fb1be7a-a4de-40ee-9805-dac41de74c30" (UID: "0fb1be7a-a4de-40ee-9805-dac41de74c30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:45.806934 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.806844 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.806934 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.806915 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwqg2\" (UniqueName: \"kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-kube-api-access-jwqg2\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.806934 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.806933 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.807184 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.806947 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fb1be7a-a4de-40ee-9805-dac41de74c30-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.851696 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.851662 5561 generic.go:356] "Generic (PLEG): container finished" podID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerID="1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438" exitCode=0 Nov 02 20:30:45.851864 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.851717 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" Nov 02 20:30:45.851864 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.851745 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" event={"ID":"9814eaca-0ca2-4925-ae97-ea136e05daff","Type":"ContainerDied","Data":"1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438"} Nov 02 20:30:45.851864 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.851801 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc" event={"ID":"9814eaca-0ca2-4925-ae97-ea136e05daff","Type":"ContainerDied","Data":"014416cd32ad27dc292261623b3ecb7c3c3e630014a9ca5ca8e47b55245c2b24"} Nov 02 20:30:45.851864 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.851830 5561 scope.go:122] "RemoveContainer" containerID="1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438" Nov 02 20:30:45.854182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.854026 5561 generic.go:356] "Generic (PLEG): container finished" podID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerID="ab333609cbdce4df06882eb4a69986f06a9c6b2b48b016aae0218aa554f7cac1" exitCode=0 Nov 02 20:30:45.854182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.854105 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" event={"ID":"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b","Type":"ContainerDied","Data":"ab333609cbdce4df06882eb4a69986f06a9c6b2b48b016aae0218aa554f7cac1"} Nov 02 20:30:45.857091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.857065 5561 generic.go:356] "Generic (PLEG): container finished" podID="0fb1be7a-a4de-40ee-9805-dac41de74c30" containerID="55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103" exitCode=0 Nov 02 20:30:45.857184 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.857128 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" Nov 02 20:30:45.857184 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.857152 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" event={"ID":"0fb1be7a-a4de-40ee-9805-dac41de74c30","Type":"ContainerDied","Data":"55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103"} Nov 02 20:30:45.857320 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.857208 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx" event={"ID":"0fb1be7a-a4de-40ee-9805-dac41de74c30","Type":"ContainerDied","Data":"57468a942ffc3d275e95968842ff52f16a3c2a15dea4638778f8cec76cb6729a"} Nov 02 20:30:45.869624 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.867256 5561 scope.go:122] "RemoveContainer" containerID="1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438" Nov 02 20:30:45.869624 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:45.867794 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438\": container with ID starting with 1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438 not found: ID does not exist" containerID="1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438" Nov 02 20:30:45.869624 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.867833 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438"} err="failed to get container status \"1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438\": rpc error: code = NotFound desc = could not find container \"1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438\": container with ID starting with 1cc59fa0b558abb659da1864ea7e3fd6d5da885d9a46af0ddf7dfa738e88e438 not found: ID does not exist" Nov 02 20:30:45.869624 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.867860 5561 scope.go:122] "RemoveContainer" containerID="55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103" Nov 02 20:30:45.882474 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.882446 5561 scope.go:122] "RemoveContainer" containerID="55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103" Nov 02 20:30:45.882800 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:30:45.882772 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103\": container with ID starting with 55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103 not found: ID does not exist" containerID="55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103" Nov 02 20:30:45.882927 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.882812 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103"} err="failed to get container status \"55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103\": rpc error: code = NotFound desc = could not find container \"55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103\": container with ID starting with 55848bc6ba756462b4cf33b3dee6c4fad0df672d482c23714657345f3cfba103 not found: ID does not exist" Nov 02 20:30:45.894761 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:45.894740 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:30:46.010157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.010121 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-podinfo\") pod \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " Nov 02 20:30:46.010382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.010179 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istiod-ca-cert\") pod \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " Nov 02 20:30:46.010382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.010226 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-token\" (UniqueName: \"kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-token\") pod \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " Nov 02 20:30:46.010382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.010278 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-envoy\") pod \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " Nov 02 20:30:46.010382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.010322 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-credential-socket\") pod \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " Nov 02 20:30:46.010605 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.010427 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-kube-api-access-l7hrj\" (UniqueName: \"kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-kube-api-access-l7hrj\") pod \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " Nov 02 20:30:46.010605 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.010526 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-certs\") pod \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " Nov 02 20:30:46.010715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.010661 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-socket\") pod \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " Nov 02 20:30:46.010715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.010689 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-data\") pod \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\" (UID: \"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b\") " Nov 02 20:30:46.011787 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.011464 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-data" pod "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" (UID: "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:46.012122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.012092 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-credential-socket" pod "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" (UID: "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:46.012369 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.012336 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc"] Nov 02 20:30:46.012452 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.012370 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istiod-ca-cert" pod "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" (UID: "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:46.012452 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.012396 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-mntxc"] Nov 02 20:30:46.012604 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.012584 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-certs" pod "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" (UID: "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:46.013094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.013070 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-socket" pod "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" (UID: "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:46.014086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.013830 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-podinfo" pod "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" (UID: "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:30:46.014992 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.014957 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-token" pod "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" (UID: "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:46.015216 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.015185 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-kube-api-access-l7hrj" pod "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" (UID: "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b"). InnerVolumeSpecName "kube-api-access-l7hrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:46.015348 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.015274 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-envoy" pod "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" (UID: "79ad491f-a6ab-40ee-8e44-e24ec9a0da3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:46.113058 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.112970 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:46.113058 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.113002 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:46.113058 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.113015 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:46.113058 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.113028 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:46.113058 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.113040 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:46.113058 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.113053 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:46.113468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.113065 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:46.113468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.113078 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7hrj\" (UniqueName: \"kubernetes.io/projected/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-kube-api-access-l7hrj\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:46.113468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.113090 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:46.116382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.116336 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx"] Nov 02 20:30:46.119697 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.119673 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-r22jx"] Nov 02 20:30:46.865203 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.865119 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" Nov 02 20:30:46.865203 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.865138 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb" event={"ID":"79ad491f-a6ab-40ee-8e44-e24ec9a0da3b","Type":"ContainerDied","Data":"ce05e84f505c0bf0cf5cae4bdb0d7f61b1169f202b5cf6f3bab9016108e54f24"} Nov 02 20:30:46.865203 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:46.865200 5561 scope.go:122] "RemoveContainer" containerID="ab333609cbdce4df06882eb4a69986f06a9c6b2b48b016aae0218aa554f7cac1" Nov 02 20:30:47.029347 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:47.024046 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb"] Nov 02 20:30:47.068707 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:47.068668 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-q5tqb"] Nov 02 20:30:47.314320 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:47.314285 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb1be7a-a4de-40ee-9805-dac41de74c30" path="/var/lib/kubelet/pods/0fb1be7a-a4de-40ee-9805-dac41de74c30/volumes" Nov 02 20:30:47.314768 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:47.314753 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" path="/var/lib/kubelet/pods/79ad491f-a6ab-40ee-8e44-e24ec9a0da3b/volumes" Nov 02 20:30:47.315208 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:47.315195 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" path="/var/lib/kubelet/pods/9814eaca-0ca2-4925-ae97-ea136e05daff/volumes" Nov 02 20:30:48.840931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:48.840900 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:30:49.171770 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:49.171736 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:53.688735 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:53.688679 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:53.939415 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:53.939307 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:55.462298 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:55.462260 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:58.177812 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:30:58.177779 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:02.765573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:02.765514 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:03.780125 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:31:03.780089 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:31:06.886775 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.886741 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl"] Nov 02 20:31:06.887509 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887490 5561 cpu_manager.go:414] "containerMap: removing container" podUID="0fb1be7a-a4de-40ee-9805-dac41de74c30" containerName="istio-proxy" Nov 02 20:31:06.887567 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887512 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb1be7a-a4de-40ee-9805-dac41de74c30" containerName="istio-proxy" Nov 02 20:31:06.887567 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887537 5561 cpu_manager.go:414] "containerMap: removing container" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" Nov 02 20:31:06.887567 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887544 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" Nov 02 20:31:06.887695 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887575 5561 cpu_manager.go:414] "containerMap: removing container" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" containerName="cleanup" Nov 02 20:31:06.887695 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887583 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f03537-f830-4843-b04a-6c2de40b6234" containerName="cleanup" Nov 02 20:31:06.887695 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887603 5561 cpu_manager.go:414] "containerMap: removing container" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" Nov 02 20:31:06.887695 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887610 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" Nov 02 20:31:06.887695 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887647 5561 cpu_manager.go:414] "containerMap: removing container" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" Nov 02 20:31:06.887695 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887655 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" Nov 02 20:31:06.887931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887762 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="9814eaca-0ca2-4925-ae97-ea136e05daff" containerName="istio-proxy" Nov 02 20:31:06.887931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887774 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca13fc0-df97-43b4-8238-38efc937ad55" containerName="cleanup" Nov 02 20:31:06.887931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887781 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb1be7a-a4de-40ee-9805-dac41de74c30" containerName="istio-proxy" Nov 02 20:31:06.887931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887788 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1780ec-55b0-41e8-bd7d-ba8f7d369f66" containerName="istio-proxy" Nov 02 20:31:06.887931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.887798 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ad491f-a6ab-40ee-8e44-e24ec9a0da3b" containerName="istio-proxy" Nov 02 20:31:06.892708 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.892690 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:06.944718 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.944687 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl"] Nov 02 20:31:06.975807 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.975765 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6q8\" (UniqueName: \"kubernetes.io/projected/de0de27e-70b6-4f5d-b269-2e1651c6dd53-kube-api-access-dw6q8\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:06.976002 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.975869 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0de27e-70b6-4f5d-b269-2e1651c6dd53-openshift-service-ca\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:06.976002 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.975932 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-maas-api-tls\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:06.976076 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.976016 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-metrics-certs\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:06.988311 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:06.988279 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-62d287-serving-cert\"" Nov 02 20:31:07.077919 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.077865 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6q8\" (UniqueName: \"kubernetes.io/projected/de0de27e-70b6-4f5d-b269-2e1651c6dd53-kube-api-access-dw6q8\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:07.078103 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.077934 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0de27e-70b6-4f5d-b269-2e1651c6dd53-openshift-service-ca\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:07.078103 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.077957 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-maas-api-tls\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:07.078103 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.077994 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-metrics-certs\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:07.078103 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:31:07.078092 5561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-serving-cert: secret "maas-api-e2e-inf-a-62d287-serving-cert" not found Nov 02 20:31:07.078315 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:31:07.078192 5561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-maas-api-tls podName:de0de27e-70b6-4f5d-b269-2e1651c6dd53 nodeName:}" failed. No retries permitted until 2026-11-02 20:31:07.578145124 +0000 UTC m=+2304.385929334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-maas-api-tls") pod "maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" (UID: "de0de27e-70b6-4f5d-b269-2e1651c6dd53") : secret "maas-api-e2e-inf-a-62d287-serving-cert" not found Nov 02 20:31:07.078740 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.078718 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0de27e-70b6-4f5d-b269-2e1651c6dd53-openshift-service-ca\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:07.080551 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.080533 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-metrics-certs\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:07.125980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.125892 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:07.132641 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.132614 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6q8\" (UniqueName: \"kubernetes.io/projected/de0de27e-70b6-4f5d-b269-2e1651c6dd53-kube-api-access-dw6q8\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:07.300430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.300386 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:07.492668 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.492627 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:07.585354 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.585270 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-maas-api-tls\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:07.587739 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.587713 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-maas-api-tls\") pod \"maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:07.806107 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:07.806070 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:08.187098 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:08.187060 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl"] Nov 02 20:31:08.194038 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:31:08.194008 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0de27e_70b6_4f5d_b269_2e1651c6dd53.slice/crio-5b338fc625b9ab753e164a1506629b13f90e311531eea0cc414476af03dffc2c WatchSource:0}: Error finding container 5b338fc625b9ab753e164a1506629b13f90e311531eea0cc414476af03dffc2c: Status 404 returned error can't find the container with id 5b338fc625b9ab753e164a1506629b13f90e311531eea0cc414476af03dffc2c Nov 02 20:31:08.195539 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:08.195519 5561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:31:09.002465 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:09.002427 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" event={"ID":"de0de27e-70b6-4f5d-b269-2e1651c6dd53","Type":"ContainerStarted","Data":"5b338fc625b9ab753e164a1506629b13f90e311531eea0cc414476af03dffc2c"} Nov 02 20:31:09.043417 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:09.043389 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:09.580822 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:09.580790 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:09.759625 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:09.759543 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:10.010709 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:10.010669 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" event={"ID":"de0de27e-70b6-4f5d-b269-2e1651c6dd53","Type":"ContainerStarted","Data":"b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9"} Nov 02 20:31:10.010952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:10.010723 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:10.163152 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:10.163091 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" podStartSLOduration=3.335223806 podStartE2EDuration="4.163077822s" totalImagesPullingTime="827.854016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:10.162568517 +0000 UTC m=+2306.970352756" watchObservedRunningTime="2026-11-02 20:31:10.163077822 +0000 UTC m=+2306.970862037" Nov 02 20:31:11.265536 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:11.265495 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:12.368459 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:12.368418 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:12.767383 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:12.767333 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:13.095196 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:13.095115 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:13.674705 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:13.674674 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:14.008090 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:14.008000 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:14.220089 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:14.220051 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:14.396168 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:14.396059 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:14.874806 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:14.874772 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:16.024103 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:16.024072 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:31:16.821852 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:16.821821 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:17.028745 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:17.028714 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:19.292134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:19.292081 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:19.598870 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:19.598784 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:19.845682 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:19.845644 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:19.854845 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:19.854773 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5"] Nov 02 20:31:19.855086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:19.855060 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" podUID="fd90ccb4-69c5-498b-b5d9-bdb093974c60" containerName="istio-proxy" containerID="cri-o://1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a" gracePeriod=30 Nov 02 20:31:20.052032 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:20.051989 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:20.439808 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:20.439771 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:21.141578 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:21.141518 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:21.313996 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:21.313956 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:21.503057 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:21.503017 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:22.599260 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:22.599227 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:22.775626 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:22.775593 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:22.879236 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:22.879146 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p"] Nov 02 20:31:22.879446 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:22.879402 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" podUID="f1487346-6dd4-4ea6-8507-bc97491d3d12" containerName="istio-proxy" containerID="cri-o://ef62c1d7e439bf561b33c7d2de9d9ade5e08e76ace63957e8f8a7aa70a76d5a2" gracePeriod=30 Nov 02 20:31:22.891133 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:22.891098 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p"] Nov 02 20:31:22.891375 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:22.891348 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" podUID="53a574dd-f668-47bc-8ac4-8b11316a1a11" containerName="istio-proxy" containerID="cri-o://050f1176158376dc866cc094b38f4dfc5a114ad66aa77c06aeaa4565517579f6" gracePeriod=30 Nov 02 20:31:22.912137 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:22.912105 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5"] Nov 02 20:31:22.912367 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:22.912344 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" podUID="df81b225-af07-4ecf-a502-2a85102530a1" containerName="istio-proxy" containerID="cri-o://abb8303282d0eefdf30225509862d7b7fee8f5b85a98f71f2f9a24a4f5de83f0" gracePeriod=30 Nov 02 20:31:22.958438 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:22.958400 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:24.464119 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:24.464079 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:25.129474 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.129408 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:31:25.144751 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.144717 5561 generic.go:356] "Generic (PLEG): container finished" podID="fd90ccb4-69c5-498b-b5d9-bdb093974c60" containerID="1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a" exitCode=0 Nov 02 20:31:25.144971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.144763 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" event={"ID":"fd90ccb4-69c5-498b-b5d9-bdb093974c60","Type":"ContainerDied","Data":"1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a"} Nov 02 20:31:25.144971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.144784 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" Nov 02 20:31:25.144971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.144831 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5" event={"ID":"fd90ccb4-69c5-498b-b5d9-bdb093974c60","Type":"ContainerDied","Data":"4e626518221d364306006dbd76d4248d83790b8192988bf07c515307d75187ef"} Nov 02 20:31:25.144971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.144853 5561 scope.go:122] "RemoveContainer" containerID="1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a" Nov 02 20:31:25.154332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.154316 5561 scope.go:122] "RemoveContainer" containerID="1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a" Nov 02 20:31:25.154599 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:31:25.154584 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a\": container with ID starting with 1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a not found: ID does not exist" containerID="1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a" Nov 02 20:31:25.154655 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.154609 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a"} err="failed to get container status \"1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a\": rpc error: code = NotFound desc = could not find container \"1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a\": container with ID starting with 1027ad3d6a7c0bf04be83406edafa71796d5dcd834262189d0879dd92bafab2a not found: ID does not exist" Nov 02 20:31:25.269024 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.268988 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-kube-api-access-z88vw\" (UniqueName: \"kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-kube-api-access-z88vw\") pod \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " Nov 02 20:31:25.269217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.269071 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-data\") pod \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " Nov 02 20:31:25.269297 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.269250 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-certs\") pod \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " Nov 02 20:31:25.269380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.269363 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istiod-ca-cert\") pod \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " Nov 02 20:31:25.269440 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.269395 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-socket\") pod \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " Nov 02 20:31:25.269440 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.269420 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-data" pod "fd90ccb4-69c5-498b-b5d9-bdb093974c60" (UID: "fd90ccb4-69c5-498b-b5d9-bdb093974c60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:25.269550 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.269443 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-podinfo\") pod \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " Nov 02 20:31:25.269550 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.269497 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-token\" (UniqueName: \"kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-token\") pod \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " Nov 02 20:31:25.269550 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.269537 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-credential-socket\") pod \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " Nov 02 20:31:25.269730 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.269566 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-envoy\") pod \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\" (UID: \"fd90ccb4-69c5-498b-b5d9-bdb093974c60\") " Nov 02 20:31:25.269730 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.269533 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-certs" pod "fd90ccb4-69c5-498b-b5d9-bdb093974c60" (UID: "fd90ccb4-69c5-498b-b5d9-bdb093974c60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:25.269730 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.269673 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-socket" pod "fd90ccb4-69c5-498b-b5d9-bdb093974c60" (UID: "fd90ccb4-69c5-498b-b5d9-bdb093974c60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:25.269939 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.269816 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-credential-socket" pod "fd90ccb4-69c5-498b-b5d9-bdb093974c60" (UID: "fd90ccb4-69c5-498b-b5d9-bdb093974c60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:25.270250 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.270221 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istiod-ca-cert" pod "fd90ccb4-69c5-498b-b5d9-bdb093974c60" (UID: "fd90ccb4-69c5-498b-b5d9-bdb093974c60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:25.270361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.270229 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.270361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.270281 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.270361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.270299 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.270361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.270313 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.271844 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.271822 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-envoy" pod "fd90ccb4-69c5-498b-b5d9-bdb093974c60" (UID: "fd90ccb4-69c5-498b-b5d9-bdb093974c60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:25.271965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.271942 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-kube-api-access-z88vw" pod "fd90ccb4-69c5-498b-b5d9-bdb093974c60" (UID: "fd90ccb4-69c5-498b-b5d9-bdb093974c60"). InnerVolumeSpecName "kube-api-access-z88vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:25.272104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.272079 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-podinfo" pod "fd90ccb4-69c5-498b-b5d9-bdb093974c60" (UID: "fd90ccb4-69c5-498b-b5d9-bdb093974c60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:25.272571 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.272549 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-token" pod "fd90ccb4-69c5-498b-b5d9-bdb093974c60" (UID: "fd90ccb4-69c5-498b-b5d9-bdb093974c60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:25.372235 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.372188 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.372235 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.372238 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.372454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.372254 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.372454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.372268 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd90ccb4-69c5-498b-b5d9-bdb093974c60-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.372454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.372283 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z88vw\" (UniqueName: \"kubernetes.io/projected/fd90ccb4-69c5-498b-b5d9-bdb093974c60-kube-api-access-z88vw\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.555023 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.554979 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5"] Nov 02 20:31:25.557642 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:25.557613 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-g8bz5"] Nov 02 20:31:27.313335 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:27.313303 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd90ccb4-69c5-498b-b5d9-bdb093974c60" path="/var/lib/kubelet/pods/fd90ccb4-69c5-498b-b5d9-bdb093974c60/volumes" Nov 02 20:31:28.166626 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.166597 5561 generic.go:356] "Generic (PLEG): container finished" podID="53a574dd-f668-47bc-8ac4-8b11316a1a11" containerID="050f1176158376dc866cc094b38f4dfc5a114ad66aa77c06aeaa4565517579f6" exitCode=0 Nov 02 20:31:28.166800 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.166681 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" event={"ID":"53a574dd-f668-47bc-8ac4-8b11316a1a11","Type":"ContainerDied","Data":"050f1176158376dc866cc094b38f4dfc5a114ad66aa77c06aeaa4565517579f6"} Nov 02 20:31:28.169060 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.168800 5561 generic.go:356] "Generic (PLEG): container finished" podID="f1487346-6dd4-4ea6-8507-bc97491d3d12" containerID="ef62c1d7e439bf561b33c7d2de9d9ade5e08e76ace63957e8f8a7aa70a76d5a2" exitCode=0 Nov 02 20:31:28.169060 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.168947 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" event={"ID":"f1487346-6dd4-4ea6-8507-bc97491d3d12","Type":"ContainerDied","Data":"ef62c1d7e439bf561b33c7d2de9d9ade5e08e76ace63957e8f8a7aa70a76d5a2"} Nov 02 20:31:28.187913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.180573 5561 generic.go:356] "Generic (PLEG): container finished" podID="df81b225-af07-4ecf-a502-2a85102530a1" containerID="abb8303282d0eefdf30225509862d7b7fee8f5b85a98f71f2f9a24a4f5de83f0" exitCode=0 Nov 02 20:31:28.187913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.180620 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" event={"ID":"df81b225-af07-4ecf-a502-2a85102530a1","Type":"ContainerDied","Data":"abb8303282d0eefdf30225509862d7b7fee8f5b85a98f71f2f9a24a4f5de83f0"} Nov 02 20:31:28.191487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.191464 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:31:28.229118 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.229083 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:31:28.248323 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.248299 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:31:28.310241 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.310205 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-token\" (UniqueName: \"kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-token\") pod \"53a574dd-f668-47bc-8ac4-8b11316a1a11\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " Nov 02 20:31:28.310424 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.310292 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-podinfo\") pod \"53a574dd-f668-47bc-8ac4-8b11316a1a11\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " Nov 02 20:31:28.310424 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.310342 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-credential-socket\") pod \"53a574dd-f668-47bc-8ac4-8b11316a1a11\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " Nov 02 20:31:28.310424 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.310379 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-socket\") pod \"53a574dd-f668-47bc-8ac4-8b11316a1a11\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " Nov 02 20:31:28.310601 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.310457 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-envoy\") pod \"53a574dd-f668-47bc-8ac4-8b11316a1a11\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " Nov 02 20:31:28.310601 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.310507 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-certs\") pod \"53a574dd-f668-47bc-8ac4-8b11316a1a11\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " Nov 02 20:31:28.310601 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.310582 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a574dd-f668-47bc-8ac4-8b11316a1a11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a574dd-f668-47bc-8ac4-8b11316a1a11-istiod-ca-cert\") pod \"53a574dd-f668-47bc-8ac4-8b11316a1a11\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " Nov 02 20:31:28.310769 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.310639 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-data\") pod \"53a574dd-f668-47bc-8ac4-8b11316a1a11\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " Nov 02 20:31:28.310769 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.310663 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-credential-socket" pod "53a574dd-f668-47bc-8ac4-8b11316a1a11" (UID: "53a574dd-f668-47bc-8ac4-8b11316a1a11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.310769 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.310695 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-kube-api-access-p655t\" (UniqueName: \"kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-kube-api-access-p655t\") pod \"53a574dd-f668-47bc-8ac4-8b11316a1a11\" (UID: \"53a574dd-f668-47bc-8ac4-8b11316a1a11\") " Nov 02 20:31:28.311016 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.310948 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-data" pod "53a574dd-f668-47bc-8ac4-8b11316a1a11" (UID: "53a574dd-f668-47bc-8ac4-8b11316a1a11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.311166 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.311142 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-certs" pod "53a574dd-f668-47bc-8ac4-8b11316a1a11" (UID: "53a574dd-f668-47bc-8ac4-8b11316a1a11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.311327 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.311302 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a574dd-f668-47bc-8ac4-8b11316a1a11-istiod-ca-cert" pod "53a574dd-f668-47bc-8ac4-8b11316a1a11" (UID: "53a574dd-f668-47bc-8ac4-8b11316a1a11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:28.311506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.311480 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a574dd-f668-47bc-8ac4-8b11316a1a11-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.311506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.311507 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.311694 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.311522 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.311694 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.311537 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.311694 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.310855 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-socket" pod "53a574dd-f668-47bc-8ac4-8b11316a1a11" (UID: "53a574dd-f668-47bc-8ac4-8b11316a1a11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.312571 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.312546 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-podinfo" pod "53a574dd-f668-47bc-8ac4-8b11316a1a11" (UID: "53a574dd-f668-47bc-8ac4-8b11316a1a11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:28.313191 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.313166 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-envoy" pod "53a574dd-f668-47bc-8ac4-8b11316a1a11" (UID: "53a574dd-f668-47bc-8ac4-8b11316a1a11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.313274 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.313194 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-token" pod "53a574dd-f668-47bc-8ac4-8b11316a1a11" (UID: "53a574dd-f668-47bc-8ac4-8b11316a1a11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:28.313274 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.313245 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-kube-api-access-p655t" pod "53a574dd-f668-47bc-8ac4-8b11316a1a11" (UID: "53a574dd-f668-47bc-8ac4-8b11316a1a11"). InnerVolumeSpecName "kube-api-access-p655t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:28.412235 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412205 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-envoy\") pod \"df81b225-af07-4ecf-a502-2a85102530a1\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " Nov 02 20:31:28.412235 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412247 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-istio-token\" (UniqueName: \"kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-istio-token\") pod \"df81b225-af07-4ecf-a502-2a85102530a1\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " Nov 02 20:31:28.412765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412278 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-data\") pod \"df81b225-af07-4ecf-a502-2a85102530a1\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " Nov 02 20:31:28.412765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412319 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-socket\") pod \"df81b225-af07-4ecf-a502-2a85102530a1\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " Nov 02 20:31:28.412765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412361 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-certs\") pod \"df81b225-af07-4ecf-a502-2a85102530a1\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " Nov 02 20:31:28.412765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412412 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-kube-api-access-r5c2w\" (UniqueName: \"kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-kube-api-access-r5c2w\") pod \"df81b225-af07-4ecf-a502-2a85102530a1\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " Nov 02 20:31:28.412765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412505 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-kube-api-access-cfbk4\" (UniqueName: \"kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-kube-api-access-cfbk4\") pod \"f1487346-6dd4-4ea6-8507-bc97491d3d12\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " Nov 02 20:31:28.412765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412552 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1487346-6dd4-4ea6-8507-bc97491d3d12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1487346-6dd4-4ea6-8507-bc97491d3d12-istiod-ca-cert\") pod \"f1487346-6dd4-4ea6-8507-bc97491d3d12\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " Nov 02 20:31:28.412765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412580 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-certs\") pod \"f1487346-6dd4-4ea6-8507-bc97491d3d12\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " Nov 02 20:31:28.412765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412616 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-token\" (UniqueName: \"kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-token\") pod \"f1487346-6dd4-4ea6-8507-bc97491d3d12\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " Nov 02 20:31:28.412765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412643 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-envoy\") pod \"f1487346-6dd4-4ea6-8507-bc97491d3d12\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " Nov 02 20:31:28.412765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412658 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-certs" pod "df81b225-af07-4ecf-a502-2a85102530a1" (UID: "df81b225-af07-4ecf-a502-2a85102530a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.412765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412678 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-socket" pod "df81b225-af07-4ecf-a502-2a85102530a1" (UID: "df81b225-af07-4ecf-a502-2a85102530a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.412765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412729 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-credential-socket\") pod \"df81b225-af07-4ecf-a502-2a85102530a1\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " Nov 02 20:31:28.413430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412912 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-podinfo\") pod \"f1487346-6dd4-4ea6-8507-bc97491d3d12\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " Nov 02 20:31:28.413430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412918 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-data" pod "df81b225-af07-4ecf-a502-2a85102530a1" (UID: "df81b225-af07-4ecf-a502-2a85102530a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.413430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412929 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-certs" pod "f1487346-6dd4-4ea6-8507-bc97491d3d12" (UID: "f1487346-6dd4-4ea6-8507-bc97491d3d12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.413430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.412956 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-socket\") pod \"f1487346-6dd4-4ea6-8507-bc97491d3d12\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " Nov 02 20:31:28.413430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413008 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-credential-socket\") pod \"f1487346-6dd4-4ea6-8507-bc97491d3d12\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " Nov 02 20:31:28.413430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413042 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df81b225-af07-4ecf-a502-2a85102530a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df81b225-af07-4ecf-a502-2a85102530a1-istiod-ca-cert\") pod \"df81b225-af07-4ecf-a502-2a85102530a1\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " Nov 02 20:31:28.413430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413069 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-data\") pod \"f1487346-6dd4-4ea6-8507-bc97491d3d12\" (UID: \"f1487346-6dd4-4ea6-8507-bc97491d3d12\") " Nov 02 20:31:28.413430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413101 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df81b225-af07-4ecf-a502-2a85102530a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df81b225-af07-4ecf-a502-2a85102530a1-istio-podinfo\") pod \"df81b225-af07-4ecf-a502-2a85102530a1\" (UID: \"df81b225-af07-4ecf-a502-2a85102530a1\") " Nov 02 20:31:28.413430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413128 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-credential-socket" pod "df81b225-af07-4ecf-a502-2a85102530a1" (UID: "df81b225-af07-4ecf-a502-2a85102530a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.413430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413249 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1487346-6dd4-4ea6-8507-bc97491d3d12-istiod-ca-cert" pod "f1487346-6dd4-4ea6-8507-bc97491d3d12" (UID: "f1487346-6dd4-4ea6-8507-bc97491d3d12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:28.414044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413724 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.414044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413750 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.414044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413767 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.414044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413780 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.414044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413793 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.414044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413807 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.414044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413820 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a574dd-f668-47bc-8ac4-8b11316a1a11-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.414044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413835 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1487346-6dd4-4ea6-8507-bc97491d3d12-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.414044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413848 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.414044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413863 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p655t\" (UniqueName: \"kubernetes.io/projected/53a574dd-f668-47bc-8ac4-8b11316a1a11-kube-api-access-p655t\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.414044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.413925 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.414652 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.414197 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-credential-socket" pod "f1487346-6dd4-4ea6-8507-bc97491d3d12" (UID: "f1487346-6dd4-4ea6-8507-bc97491d3d12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.414652 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.414377 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df81b225-af07-4ecf-a502-2a85102530a1-istiod-ca-cert" pod "df81b225-af07-4ecf-a502-2a85102530a1" (UID: "df81b225-af07-4ecf-a502-2a85102530a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:28.414858 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.414821 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-data" pod "f1487346-6dd4-4ea6-8507-bc97491d3d12" (UID: "f1487346-6dd4-4ea6-8507-bc97491d3d12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.415083 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.415015 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-socket" pod "f1487346-6dd4-4ea6-8507-bc97491d3d12" (UID: "f1487346-6dd4-4ea6-8507-bc97491d3d12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.415317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.415279 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-token" pod "f1487346-6dd4-4ea6-8507-bc97491d3d12" (UID: "f1487346-6dd4-4ea6-8507-bc97491d3d12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:28.415612 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.415588 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df81b225-af07-4ecf-a502-2a85102530a1-istio-podinfo" pod "df81b225-af07-4ecf-a502-2a85102530a1" (UID: "df81b225-af07-4ecf-a502-2a85102530a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:28.416435 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.416408 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-istio-token" pod "df81b225-af07-4ecf-a502-2a85102530a1" (UID: "df81b225-af07-4ecf-a502-2a85102530a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:28.416728 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.416705 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-podinfo" pod "f1487346-6dd4-4ea6-8507-bc97491d3d12" (UID: "f1487346-6dd4-4ea6-8507-bc97491d3d12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:28.417206 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.417180 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-envoy" pod "df81b225-af07-4ecf-a502-2a85102530a1" (UID: "df81b225-af07-4ecf-a502-2a85102530a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.417561 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.417529 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-kube-api-access-r5c2w" pod "df81b225-af07-4ecf-a502-2a85102530a1" (UID: "df81b225-af07-4ecf-a502-2a85102530a1"). InnerVolumeSpecName "kube-api-access-r5c2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:28.417999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.417981 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-kube-api-access-cfbk4" pod "f1487346-6dd4-4ea6-8507-bc97491d3d12" (UID: "f1487346-6dd4-4ea6-8507-bc97491d3d12"). InnerVolumeSpecName "kube-api-access-cfbk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:28.418343 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.418329 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-envoy" pod "f1487346-6dd4-4ea6-8507-bc97491d3d12" (UID: "f1487346-6dd4-4ea6-8507-bc97491d3d12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.515156 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.515130 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.515156 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.515153 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.515156 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.515163 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df81b225-af07-4ecf-a502-2a85102530a1-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.515410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.515175 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.515410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.515188 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df81b225-af07-4ecf-a502-2a85102530a1-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.515410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.515200 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df81b225-af07-4ecf-a502-2a85102530a1-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.515410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.515212 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.515410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.515224 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5c2w\" (UniqueName: \"kubernetes.io/projected/df81b225-af07-4ecf-a502-2a85102530a1-kube-api-access-r5c2w\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.515410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.515241 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfbk4\" (UniqueName: \"kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-kube-api-access-cfbk4\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.515410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.515253 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.515410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.515265 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.515410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:28.515277 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1487346-6dd4-4ea6-8507-bc97491d3d12-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:29.196911 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.196787 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" Nov 02 20:31:29.196911 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.196797 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p" event={"ID":"f1487346-6dd4-4ea6-8507-bc97491d3d12","Type":"ContainerDied","Data":"b654c72bbfeaff7d39389554f27e5d2df1b840e31794201f08f3d0977b4dd4b4"} Nov 02 20:31:29.197156 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.196925 5561 scope.go:122] "RemoveContainer" containerID="ef62c1d7e439bf561b33c7d2de9d9ade5e08e76ace63957e8f8a7aa70a76d5a2" Nov 02 20:31:29.198566 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.198537 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" event={"ID":"df81b225-af07-4ecf-a502-2a85102530a1","Type":"ContainerDied","Data":"cd3a4bd78795bfa2cdaf0f62be7287c58a64388c6e6a38ac9dca5ca53534d971"} Nov 02 20:31:29.198689 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.198550 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5" Nov 02 20:31:29.200272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.200251 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" event={"ID":"53a574dd-f668-47bc-8ac4-8b11316a1a11","Type":"ContainerDied","Data":"d7ffb8ac2fe64d4fc4a43edf6d9276b0388d0c4141f70521eb843fdbfd480f29"} Nov 02 20:31:29.200377 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.200297 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p" Nov 02 20:31:29.209286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.209263 5561 scope.go:122] "RemoveContainer" containerID="abb8303282d0eefdf30225509862d7b7fee8f5b85a98f71f2f9a24a4f5de83f0" Nov 02 20:31:29.222236 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.222218 5561 scope.go:122] "RemoveContainer" containerID="050f1176158376dc866cc094b38f4dfc5a114ad66aa77c06aeaa4565517579f6" Nov 02 20:31:29.353500 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.353463 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5"] Nov 02 20:31:29.354643 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.354617 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:29.400889 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.400839 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-rg4t5"] Nov 02 20:31:29.461513 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.461430 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p"] Nov 02 20:31:29.466005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.465978 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-pq56p"] Nov 02 20:31:29.569536 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.569499 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p"] Nov 02 20:31:29.571529 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:29.571500 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-wfq7p"] Nov 02 20:31:31.313548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:31.313517 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a574dd-f668-47bc-8ac4-8b11316a1a11" path="/var/lib/kubelet/pods/53a574dd-f668-47bc-8ac4-8b11316a1a11/volumes" Nov 02 20:31:31.314101 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:31.314083 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df81b225-af07-4ecf-a502-2a85102530a1" path="/var/lib/kubelet/pods/df81b225-af07-4ecf-a502-2a85102530a1/volumes" Nov 02 20:31:31.314567 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:31.314553 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1487346-6dd4-4ea6-8507-bc97491d3d12" path="/var/lib/kubelet/pods/f1487346-6dd4-4ea6-8507-bc97491d3d12/volumes" Nov 02 20:31:32.639395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:32.639359 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:32.836803 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:32.836762 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:33.782955 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:31:33.782848 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:31:34.372332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:34.372296 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:35.155834 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:35.155799 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:35.442259 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:35.442146 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:39.106240 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:39.106202 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:39.593436 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:39.593400 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:39.762331 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:39.762297 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:40.792971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:40.792941 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:43.540104 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.540073 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch"] Nov 02 20:31:43.540530 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.540278 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" podUID="ec07656a-a444-401f-81f1-b229a22f4e13" containerName="maas-api" containerID="cri-o://ae2a71ea411b05fefcae16cbd3bab47c191e80712e48e35b62c58985f3485a24" gracePeriod=30 Nov 02 20:31:43.683185 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.683150 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:43.827269 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.827239 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:31:43.900677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.900642 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec07656a-a444-401f-81f1-b229a22f4e13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec07656a-a444-401f-81f1-b229a22f4e13-openshift-service-ca\") pod \"ec07656a-a444-401f-81f1-b229a22f4e13\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " Nov 02 20:31:43.900914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.900765 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-metrics-certs\") pod \"ec07656a-a444-401f-81f1-b229a22f4e13\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " Nov 02 20:31:43.900914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.900803 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-maas-api-tls\") pod \"ec07656a-a444-401f-81f1-b229a22f4e13\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " Nov 02 20:31:43.901052 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.900960 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec07656a-a444-401f-81f1-b229a22f4e13-kube-api-access-vmjxw\" (UniqueName: \"kubernetes.io/projected/ec07656a-a444-401f-81f1-b229a22f4e13-kube-api-access-vmjxw\") pod \"ec07656a-a444-401f-81f1-b229a22f4e13\" (UID: \"ec07656a-a444-401f-81f1-b229a22f4e13\") " Nov 02 20:31:43.901137 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.901105 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec07656a-a444-401f-81f1-b229a22f4e13-openshift-service-ca" pod "ec07656a-a444-401f-81f1-b229a22f4e13" (UID: "ec07656a-a444-401f-81f1-b229a22f4e13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:43.901554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.901530 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec07656a-a444-401f-81f1-b229a22f4e13-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.903772 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.903740 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-maas-api-tls" pod "ec07656a-a444-401f-81f1-b229a22f4e13" (UID: "ec07656a-a444-401f-81f1-b229a22f4e13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:43.903772 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.903765 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-metrics-certs" pod "ec07656a-a444-401f-81f1-b229a22f4e13" (UID: "ec07656a-a444-401f-81f1-b229a22f4e13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:43.908024 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:43.907998 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec07656a-a444-401f-81f1-b229a22f4e13-kube-api-access-vmjxw" pod "ec07656a-a444-401f-81f1-b229a22f4e13" (UID: "ec07656a-a444-401f-81f1-b229a22f4e13"). InnerVolumeSpecName "kube-api-access-vmjxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:44.003195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:44.003162 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:44.003195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:44.003188 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec07656a-a444-401f-81f1-b229a22f4e13-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:44.003195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:44.003198 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjxw\" (UniqueName: \"kubernetes.io/projected/ec07656a-a444-401f-81f1-b229a22f4e13-kube-api-access-vmjxw\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:44.044646 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:44.044188 5561 scope.go:122] "RemoveContainer" containerID="ae2a71ea411b05fefcae16cbd3bab47c191e80712e48e35b62c58985f3485a24" Nov 02 20:31:44.293963 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:44.293919 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" event={"ID":"ec07656a-a444-401f-81f1-b229a22f4e13","Type":"ContainerDied","Data":"ae2a71ea411b05fefcae16cbd3bab47c191e80712e48e35b62c58985f3485a24"} Nov 02 20:31:44.294162 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:44.293991 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" Nov 02 20:31:44.294162 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:44.293990 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch" event={"ID":"ec07656a-a444-401f-81f1-b229a22f4e13","Type":"ContainerDied","Data":"52c488f75b027d49656ee94bda5ae1b9b2788dbafc69c7a1c5a8afc3ced52447"} Nov 02 20:31:44.528895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:44.528676 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch"] Nov 02 20:31:44.543901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:44.543546 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be1348-5b6c875564-nb7ch"] Nov 02 20:31:45.314295 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:45.314261 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec07656a-a444-401f-81f1-b229a22f4e13" path="/var/lib/kubelet/pods/ec07656a-a444-401f-81f1-b229a22f4e13/volumes" Nov 02 20:31:45.721310 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:45.721275 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm"] Nov 02 20:31:45.768635 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:45.768596 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt"] Nov 02 20:31:46.222707 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:46.222676 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:50.315378 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:50.315335 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:51.987638 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:51.987602 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:52.173228 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:52.173191 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:52.382961 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:52.382830 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:53.276948 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:53.276913 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:53.449537 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:53.449499 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:54.286158 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.286075 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:54.973426 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.973391 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52"] Nov 02 20:31:54.973962 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.973938 5561 cpu_manager.go:414] "containerMap: removing container" podUID="53a574dd-f668-47bc-8ac4-8b11316a1a11" containerName="istio-proxy" Nov 02 20:31:54.973962 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.973958 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a574dd-f668-47bc-8ac4-8b11316a1a11" containerName="istio-proxy" Nov 02 20:31:54.974164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.973977 5561 cpu_manager.go:414] "containerMap: removing container" podUID="ec07656a-a444-401f-81f1-b229a22f4e13" containerName="maas-api" Nov 02 20:31:54.974164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.973985 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec07656a-a444-401f-81f1-b229a22f4e13" containerName="maas-api" Nov 02 20:31:54.974164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.973993 5561 cpu_manager.go:414] "containerMap: removing container" podUID="df81b225-af07-4ecf-a502-2a85102530a1" containerName="istio-proxy" Nov 02 20:31:54.974164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.973999 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df81b225-af07-4ecf-a502-2a85102530a1" containerName="istio-proxy" Nov 02 20:31:54.974164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.974021 5561 cpu_manager.go:414] "containerMap: removing container" podUID="f1487346-6dd4-4ea6-8507-bc97491d3d12" containerName="istio-proxy" Nov 02 20:31:54.974164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.974026 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1487346-6dd4-4ea6-8507-bc97491d3d12" containerName="istio-proxy" Nov 02 20:31:54.974164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.974034 5561 cpu_manager.go:414] "containerMap: removing container" podUID="fd90ccb4-69c5-498b-b5d9-bdb093974c60" containerName="istio-proxy" Nov 02 20:31:54.974164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.974038 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd90ccb4-69c5-498b-b5d9-bdb093974c60" containerName="istio-proxy" Nov 02 20:31:54.974164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.974134 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1487346-6dd4-4ea6-8507-bc97491d3d12" containerName="istio-proxy" Nov 02 20:31:54.974164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.974146 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd90ccb4-69c5-498b-b5d9-bdb093974c60" containerName="istio-proxy" Nov 02 20:31:54.974164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.974161 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a574dd-f668-47bc-8ac4-8b11316a1a11" containerName="istio-proxy" Nov 02 20:31:54.974164 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.974171 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec07656a-a444-401f-81f1-b229a22f4e13" containerName="maas-api" Nov 02 20:31:54.974659 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.974178 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="df81b225-af07-4ecf-a502-2a85102530a1" containerName="istio-proxy" Nov 02 20:31:54.979743 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:54.979720 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.029355 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.029318 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52"] Nov 02 20:31:55.074445 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.074391 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bb78ba-serving-cert\"" Nov 02 20:31:55.160703 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.160654 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-maas-api-tls\") pod \"maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.160703 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.160707 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.161007 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.160978 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-metrics-certs\") pod \"maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.161137 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.161116 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24x2\" (UniqueName: \"kubernetes.io/projected/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-kube-api-access-z24x2\") pod \"maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.212074 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.212045 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:55.262039 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.261957 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-metrics-certs\") pod \"maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.262216 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.262071 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24x2\" (UniqueName: \"kubernetes.io/projected/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-kube-api-access-z24x2\") pod \"maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.262216 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.262153 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-maas-api-tls\") pod \"maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.262216 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.262183 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.263005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.262980 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.265219 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.265194 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-metrics-certs\") pod \"maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.265415 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.265390 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-maas-api-tls\") pod \"maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.315939 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.315901 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24x2\" (UniqueName: \"kubernetes.io/projected/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-kube-api-access-z24x2\") pod \"maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.397511 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.397477 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:55.570888 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.570774 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:55.593636 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.593594 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:55.972819 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:55.972780 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52"] Nov 02 20:31:55.979890 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:31:55.979854 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ab9f5e_0844_477c_ab31_58dfc7df6d7d.slice/crio-4fb7509478228728595c878a245b6fcccac9b0814e69e8bb19534a750f4b6a74 WatchSource:0}: Error finding container 4fb7509478228728595c878a245b6fcccac9b0814e69e8bb19534a750f4b6a74: Status 404 returned error can't find the container with id 4fb7509478228728595c878a245b6fcccac9b0814e69e8bb19534a750f4b6a74 Nov 02 20:31:56.376649 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:56.376551 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" event={"ID":"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d","Type":"ContainerStarted","Data":"4fb7509478228728595c878a245b6fcccac9b0814e69e8bb19534a750f4b6a74"} Nov 02 20:31:57.384262 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:57.384218 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" event={"ID":"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d","Type":"ContainerStarted","Data":"18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b"} Nov 02 20:31:57.384744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:57.384288 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:31:57.494122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:31:57.494052 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" podStartSLOduration=2.615799453 podStartE2EDuration="3.49403166s" totalImagesPullingTime="878.232207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:57.492010768 +0000 UTC m=+2354.299795015" watchObservedRunningTime="2026-11-02 20:31:57.49403166 +0000 UTC m=+2354.301815877" Nov 02 20:32:00.344971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:00.344925 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:00.741846 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:00.741803 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" containerID="cri-o://8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954" gracePeriod=45 Nov 02 20:32:00.788160 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:00.788123 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" containerID="cri-o://81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a" gracePeriod=45 Nov 02 20:32:01.045028 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.044998 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:32:01.078273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.078247 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:32:01.130202 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130170 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-home\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-home\") pod \"390dcf22-a018-4320-986a-9b45f5958a70\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " Nov 02 20:32:01.130444 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130211 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724fade4-5d2f-4d3d-9952-351ab691d309-kube-api-access-hj2wj\" (UniqueName: \"kubernetes.io/projected/724fade4-5d2f-4d3d-9952-351ab691d309-kube-api-access-hj2wj\") pod \"724fade4-5d2f-4d3d-9952-351ab691d309\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " Nov 02 20:32:01.130444 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130258 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390dcf22-a018-4320-986a-9b45f5958a70-kube-api-access-28z7p\" (UniqueName: \"kubernetes.io/projected/390dcf22-a018-4320-986a-9b45f5958a70-kube-api-access-28z7p\") pod \"390dcf22-a018-4320-986a-9b45f5958a70\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " Nov 02 20:32:01.130444 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130287 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/724fade4-5d2f-4d3d-9952-351ab691d309-tls-certs\" (UniqueName: \"kubernetes.io/secret/724fade4-5d2f-4d3d-9952-351ab691d309-tls-certs\") pod \"724fade4-5d2f-4d3d-9952-351ab691d309\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " Nov 02 20:32:01.130444 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130340 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-tmp-dir\") pod \"724fade4-5d2f-4d3d-9952-351ab691d309\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " Nov 02 20:32:01.130444 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130410 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-tmp-dir\") pod \"390dcf22-a018-4320-986a-9b45f5958a70\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " Nov 02 20:32:01.130780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130453 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-model-cache\") pod \"724fade4-5d2f-4d3d-9952-351ab691d309\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " Nov 02 20:32:01.130780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130451 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-home" pod "390dcf22-a018-4320-986a-9b45f5958a70" (UID: "390dcf22-a018-4320-986a-9b45f5958a70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:01.130780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130498 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-dshm\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-dshm\") pod \"390dcf22-a018-4320-986a-9b45f5958a70\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " Nov 02 20:32:01.130780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130529 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/390dcf22-a018-4320-986a-9b45f5958a70-tls-certs\" (UniqueName: \"kubernetes.io/secret/390dcf22-a018-4320-986a-9b45f5958a70-tls-certs\") pod \"390dcf22-a018-4320-986a-9b45f5958a70\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " Nov 02 20:32:01.130780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130590 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-dshm\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-dshm\") pod \"724fade4-5d2f-4d3d-9952-351ab691d309\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " Nov 02 20:32:01.130780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130757 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-model-cache\") pod \"390dcf22-a018-4320-986a-9b45f5958a70\" (UID: \"390dcf22-a018-4320-986a-9b45f5958a70\") " Nov 02 20:32:01.131126 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130806 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-model-cache" pod "724fade4-5d2f-4d3d-9952-351ab691d309" (UID: "724fade4-5d2f-4d3d-9952-351ab691d309"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:01.131126 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.130819 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-home\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-home\") pod \"724fade4-5d2f-4d3d-9952-351ab691d309\" (UID: \"724fade4-5d2f-4d3d-9952-351ab691d309\") " Nov 02 20:32:01.131126 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.131006 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-tmp-dir" pod "390dcf22-a018-4320-986a-9b45f5958a70" (UID: "390dcf22-a018-4320-986a-9b45f5958a70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:01.131321 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.131301 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.131382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.131328 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.131382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.131345 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.131382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.131354 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-model-cache" pod "390dcf22-a018-4320-986a-9b45f5958a70" (UID: "390dcf22-a018-4320-986a-9b45f5958a70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:01.131550 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.131531 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-home" pod "724fade4-5d2f-4d3d-9952-351ab691d309" (UID: "724fade4-5d2f-4d3d-9952-351ab691d309"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:01.131618 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.131582 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-tmp-dir" pod "724fade4-5d2f-4d3d-9952-351ab691d309" (UID: "724fade4-5d2f-4d3d-9952-351ab691d309"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:01.132923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.132897 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724fade4-5d2f-4d3d-9952-351ab691d309-tls-certs" pod "724fade4-5d2f-4d3d-9952-351ab691d309" (UID: "724fade4-5d2f-4d3d-9952-351ab691d309"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:01.133235 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.133207 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390dcf22-a018-4320-986a-9b45f5958a70-kube-api-access-28z7p" pod "390dcf22-a018-4320-986a-9b45f5958a70" (UID: "390dcf22-a018-4320-986a-9b45f5958a70"). InnerVolumeSpecName "kube-api-access-28z7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:01.133408 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.133385 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724fade4-5d2f-4d3d-9952-351ab691d309-kube-api-access-hj2wj" pod "724fade4-5d2f-4d3d-9952-351ab691d309" (UID: "724fade4-5d2f-4d3d-9952-351ab691d309"). InnerVolumeSpecName "kube-api-access-hj2wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:01.133551 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.133528 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-dshm" pod "390dcf22-a018-4320-986a-9b45f5958a70" (UID: "390dcf22-a018-4320-986a-9b45f5958a70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:01.133767 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.133746 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390dcf22-a018-4320-986a-9b45f5958a70-tls-certs" pod "390dcf22-a018-4320-986a-9b45f5958a70" (UID: "390dcf22-a018-4320-986a-9b45f5958a70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:01.134446 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.134420 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-dshm" pod "724fade4-5d2f-4d3d-9952-351ab691d309" (UID: "724fade4-5d2f-4d3d-9952-351ab691d309"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:01.234318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.234283 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.234318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.234319 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390dcf22-a018-4320-986a-9b45f5958a70-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.234576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.234330 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.234576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.234338 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390dcf22-a018-4320-986a-9b45f5958a70-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.234576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.234347 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.234576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.234358 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj2wj\" (UniqueName: \"kubernetes.io/projected/724fade4-5d2f-4d3d-9952-351ab691d309-kube-api-access-hj2wj\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.234576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.234367 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28z7p\" (UniqueName: \"kubernetes.io/projected/390dcf22-a018-4320-986a-9b45f5958a70-kube-api-access-28z7p\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.234576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.234377 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/724fade4-5d2f-4d3d-9952-351ab691d309-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.234576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.234385 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/724fade4-5d2f-4d3d-9952-351ab691d309-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.410182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.410145 5561 generic.go:356] "Generic (PLEG): container finished" podID="390dcf22-a018-4320-986a-9b45f5958a70" containerID="8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954" exitCode=0 Nov 02 20:32:01.410677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.410205 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" Nov 02 20:32:01.410677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.410226 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" event={"ID":"390dcf22-a018-4320-986a-9b45f5958a70","Type":"ContainerDied","Data":"8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954"} Nov 02 20:32:01.410677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.410269 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm" event={"ID":"390dcf22-a018-4320-986a-9b45f5958a70","Type":"ContainerDied","Data":"31d0ca1f18099c3de3cc8dc5ac6748f8040e7be7edbd8776d854636080aeead5"} Nov 02 20:32:01.410677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.410287 5561 scope.go:122] "RemoveContainer" containerID="8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954" Nov 02 20:32:01.411584 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.411556 5561 generic.go:356] "Generic (PLEG): container finished" podID="724fade4-5d2f-4d3d-9952-351ab691d309" containerID="81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a" exitCode=0 Nov 02 20:32:01.411711 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.411610 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" Nov 02 20:32:01.411711 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.411644 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" event={"ID":"724fade4-5d2f-4d3d-9952-351ab691d309","Type":"ContainerDied","Data":"81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a"} Nov 02 20:32:01.411711 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.411678 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt" event={"ID":"724fade4-5d2f-4d3d-9952-351ab691d309","Type":"ContainerDied","Data":"189a1e075dd2e21ee88006996ad0bd604e2b5a8f796d29150e7f83af263adeb9"} Nov 02 20:32:01.419321 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.419288 5561 scope.go:122] "RemoveContainer" containerID="8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954" Nov 02 20:32:01.419599 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:32:01.419578 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954\": container with ID starting with 8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954 not found: ID does not exist" containerID="8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954" Nov 02 20:32:01.419664 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.419607 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954"} err="failed to get container status \"8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954\": rpc error: code = NotFound desc = could not find container \"8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954\": container with ID starting with 8c9f55cd8794835ca02d2829b0a76d37611e2fc834bea644a96e93bfc6a4e954 not found: ID does not exist" Nov 02 20:32:01.419664 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.419624 5561 scope.go:122] "RemoveContainer" containerID="81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a" Nov 02 20:32:01.427666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.427645 5561 scope.go:122] "RemoveContainer" containerID="81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a" Nov 02 20:32:01.427973 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:32:01.427956 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a\": container with ID starting with 81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a not found: ID does not exist" containerID="81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a" Nov 02 20:32:01.428027 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.427980 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a"} err="failed to get container status \"81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a\": rpc error: code = NotFound desc = could not find container \"81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a\": container with ID starting with 81f4f2da37170e776bb513917c7b81d1c7adb8f44071c9efea3a35d653109b6a not found: ID does not exist" Nov 02 20:32:01.566140 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.565944 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt"] Nov 02 20:32:01.568771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.568746 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/simulator-w2-9fcedd-kserve-5745db98fb-vk5rt"] Nov 02 20:32:01.671607 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.671574 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm"] Nov 02 20:32:01.677711 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:01.677675 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/premium-w2-9fcedd-kserve-7cb5cfd577-sjhzm"] Nov 02 20:32:03.314846 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:03.314817 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390dcf22-a018-4320-986a-9b45f5958a70" path="/var/lib/kubelet/pods/390dcf22-a018-4320-986a-9b45f5958a70/volumes" Nov 02 20:32:03.315412 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:03.315395 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" path="/var/lib/kubelet/pods/724fade4-5d2f-4d3d-9952-351ab691d309/volumes" Nov 02 20:32:03.396075 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:03.396044 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:32:03.779680 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:32:03.779639 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:32:05.087029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:05.086980 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:05.262825 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:05.262785 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:05.437669 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:05.437634 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:10.277476 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:10.277441 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:11.488760 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:11.488717 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:11.672090 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:11.672056 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:15.389586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:15.389543 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:15.725007 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:15.724943 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:15.907103 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:15.907060 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:16.593922 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:16.593850 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:20.566776 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:20.566738 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:22.097376 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:22.097339 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:23.822521 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:23.822480 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:25.665173 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:25.665140 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:26.953413 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:26.953340 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:27.153013 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:27.152979 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:28.178712 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:28.178673 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:28.421631 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:28.421595 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:30.621916 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:30.621850 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:33.779431 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:32:33.779392 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:32:35.165796 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:35.165765 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:39.758292 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:39.758251 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:44.497587 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:44.497547 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:48.829634 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:48.829597 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:50.360500 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:50.360458 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:50.554995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:50.554959 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:50.728028 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:50.727992 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:51.278129 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:51.278093 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:54.640449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:54.640410 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:55.026487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:55.026451 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:55.218171 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:55.218134 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:55.412821 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:55.412793 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:57.310755 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:57.310712 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:57.484412 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:57.484373 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:57.693916 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:57.693867 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:58.508099 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:58.508061 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:59.124292 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:59.124254 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:59.334053 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:59.334017 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:59.997675 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:32:59.997645 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:03.316293 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:03.316260 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:03.506690 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:03.506654 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:03.779154 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:33:03.779119 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:33:04.814141 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:04.814103 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:08.667638 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:08.667602 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:08.849374 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:08.849342 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:09.126489 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:09.126402 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:09.583046 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:09.583012 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:09.752448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:09.752419 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:10.386597 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:10.386560 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc"] Nov 02 20:33:10.387123 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:10.387084 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" containerID="cri-o://5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc" gracePeriod=30 Nov 02 20:33:10.466869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:10.466835 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:10.724265 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:10.724223 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:14.095792 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:14.095753 5561 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:33:14.096340 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:14.095813 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:33:14.332659 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:14.332618 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:14.525612 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:14.525579 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:14.712650 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:14.712620 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:15.650967 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.650907 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:33:15.757115 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757079 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df0c17b8-8501-45ac-8667-4a24b770b638-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df0c17b8-8501-45ac-8667-4a24b770b638-istio-podinfo\") pod \"df0c17b8-8501-45ac-8667-4a24b770b638\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " Nov 02 20:33:15.757296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757154 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-certs\") pod \"df0c17b8-8501-45ac-8667-4a24b770b638\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " Nov 02 20:33:15.757296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757178 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-credential-socket\") pod \"df0c17b8-8501-45ac-8667-4a24b770b638\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " Nov 02 20:33:15.757296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757206 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-envoy\") pod \"df0c17b8-8501-45ac-8667-4a24b770b638\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " Nov 02 20:33:15.757296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757229 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-data\") pod \"df0c17b8-8501-45ac-8667-4a24b770b638\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " Nov 02 20:33:15.757517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757305 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-kube-api-access-kzlkt\" (UniqueName: \"kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-kube-api-access-kzlkt\") pod \"df0c17b8-8501-45ac-8667-4a24b770b638\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " Nov 02 20:33:15.757517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757346 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-socket\") pod \"df0c17b8-8501-45ac-8667-4a24b770b638\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " Nov 02 20:33:15.757517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757387 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-istio-token\" (UniqueName: \"kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-istio-token\") pod \"df0c17b8-8501-45ac-8667-4a24b770b638\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " Nov 02 20:33:15.757517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757440 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0c17b8-8501-45ac-8667-4a24b770b638-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df0c17b8-8501-45ac-8667-4a24b770b638-istiod-ca-cert\") pod \"df0c17b8-8501-45ac-8667-4a24b770b638\" (UID: \"df0c17b8-8501-45ac-8667-4a24b770b638\") " Nov 02 20:33:15.757517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757492 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-certs" pod "df0c17b8-8501-45ac-8667-4a24b770b638" (UID: "df0c17b8-8501-45ac-8667-4a24b770b638"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:15.757804 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757548 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-credential-socket" pod "df0c17b8-8501-45ac-8667-4a24b770b638" (UID: "df0c17b8-8501-45ac-8667-4a24b770b638"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:15.757863 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757745 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-data" pod "df0c17b8-8501-45ac-8667-4a24b770b638" (UID: "df0c17b8-8501-45ac-8667-4a24b770b638"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:15.757863 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757801 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-socket" pod "df0c17b8-8501-45ac-8667-4a24b770b638" (UID: "df0c17b8-8501-45ac-8667-4a24b770b638"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:15.757998 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757938 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0c17b8-8501-45ac-8667-4a24b770b638-istiod-ca-cert" pod "df0c17b8-8501-45ac-8667-4a24b770b638" (UID: "df0c17b8-8501-45ac-8667-4a24b770b638"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:15.758059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.757997 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:15.758059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.758020 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:15.758059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.758033 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:15.758059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.758048 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:15.759474 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.759452 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-envoy" pod "df0c17b8-8501-45ac-8667-4a24b770b638" (UID: "df0c17b8-8501-45ac-8667-4a24b770b638"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:15.759688 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.759663 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-istio-token" pod "df0c17b8-8501-45ac-8667-4a24b770b638" (UID: "df0c17b8-8501-45ac-8667-4a24b770b638"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:15.759773 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.759742 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df0c17b8-8501-45ac-8667-4a24b770b638-istio-podinfo" pod "df0c17b8-8501-45ac-8667-4a24b770b638" (UID: "df0c17b8-8501-45ac-8667-4a24b770b638"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:33:15.759822 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.759768 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-kube-api-access-kzlkt" pod "df0c17b8-8501-45ac-8667-4a24b770b638" (UID: "df0c17b8-8501-45ac-8667-4a24b770b638"). InnerVolumeSpecName "kube-api-access-kzlkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:15.859548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.859515 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df0c17b8-8501-45ac-8667-4a24b770b638-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:15.859548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.859544 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzlkt\" (UniqueName: \"kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-kube-api-access-kzlkt\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:15.859548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.859554 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df0c17b8-8501-45ac-8667-4a24b770b638-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:15.859781 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.859562 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df0c17b8-8501-45ac-8667-4a24b770b638-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:15.859781 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.859571 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df0c17b8-8501-45ac-8667-4a24b770b638-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:15.892357 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.892323 5561 generic.go:356] "Generic (PLEG): container finished" podID="df0c17b8-8501-45ac-8667-4a24b770b638" containerID="5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc" exitCode=0 Nov 02 20:33:15.892516 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.892369 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" Nov 02 20:33:15.892516 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.892372 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" event={"ID":"df0c17b8-8501-45ac-8667-4a24b770b638","Type":"ContainerDied","Data":"5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc"} Nov 02 20:33:15.892516 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.892495 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc" event={"ID":"df0c17b8-8501-45ac-8667-4a24b770b638","Type":"ContainerDied","Data":"6dd5c93320940dca0e2e41dfcdc094af6fa31bfa06d6776f0a2dbc9e896c2cb3"} Nov 02 20:33:15.892621 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.892527 5561 scope.go:122] "RemoveContainer" containerID="5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc" Nov 02 20:33:15.903359 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.903342 5561 scope.go:122] "RemoveContainer" containerID="5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc" Nov 02 20:33:15.903636 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:33:15.903620 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc\": container with ID starting with 5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc not found: ID does not exist" containerID="5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc" Nov 02 20:33:15.903701 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:15.903648 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc"} err="failed to get container status \"5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc\": rpc error: code = NotFound desc = could not find container \"5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc\": container with ID starting with 5b8e917e8128c143f3c963a3315e277ac7189f76ff21741dab787400263dddcc not found: ID does not exist" Nov 02 20:33:16.050625 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:16.050588 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc"] Nov 02 20:33:16.060923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:16.060181 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-6wsjc"] Nov 02 20:33:17.313678 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:17.313640 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" path="/var/lib/kubelet/pods/df0c17b8-8501-45ac-8667-4a24b770b638/volumes" Nov 02 20:33:17.651081 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:17.650999 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:17.827198 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:17.827167 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:18.023008 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.022976 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg"] Nov 02 20:33:18.023458 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.023445 5561 cpu_manager.go:414] "containerMap: removing container" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" Nov 02 20:33:18.023506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.023460 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" Nov 02 20:33:18.023506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.023476 5561 cpu_manager.go:414] "containerMap: removing container" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" Nov 02 20:33:18.023506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.023481 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" Nov 02 20:33:18.023506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.023497 5561 cpu_manager.go:414] "containerMap: removing container" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" Nov 02 20:33:18.023506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.023503 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" Nov 02 20:33:18.023664 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.023594 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="724fade4-5d2f-4d3d-9952-351ab691d309" containerName="main" Nov 02 20:33:18.023664 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.023602 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="390dcf22-a018-4320-986a-9b45f5958a70" containerName="main" Nov 02 20:33:18.023664 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.023611 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0c17b8-8501-45ac-8667-4a24b770b638" containerName="istio-proxy" Nov 02 20:33:18.028369 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.028349 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" Nov 02 20:33:18.080267 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.080221 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg"] Nov 02 20:33:18.086961 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.086929 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d9zn\" (UniqueName: \"kubernetes.io/projected/b90f9ab7-7575-4bd8-b55b-93872c815747-kube-api-access-9d9zn\") pod \"payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg\" (UID: \"b90f9ab7-7575-4bd8-b55b-93872c815747\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" Nov 02 20:33:18.087227 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.087203 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90f9ab7-7575-4bd8-b55b-93872c815747-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg\" (UID: \"b90f9ab7-7575-4bd8-b55b-93872c815747\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" Nov 02 20:33:18.125680 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.125639 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7b1839\"" Nov 02 20:33:18.125834 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.125692 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7b1839-dockercfg-qkgwp\"" Nov 02 20:33:18.177392 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.177355 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll"] Nov 02 20:33:18.184172 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.184138 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" Nov 02 20:33:18.188417 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.188383 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90f9ab7-7575-4bd8-b55b-93872c815747-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg\" (UID: \"b90f9ab7-7575-4bd8-b55b-93872c815747\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" Nov 02 20:33:18.188574 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.188473 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d9zn\" (UniqueName: \"kubernetes.io/projected/b90f9ab7-7575-4bd8-b55b-93872c815747-kube-api-access-9d9zn\") pod \"payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg\" (UID: \"b90f9ab7-7575-4bd8-b55b-93872c815747\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" Nov 02 20:33:18.190311 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.189406 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90f9ab7-7575-4bd8-b55b-93872c815747-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg\" (UID: \"b90f9ab7-7575-4bd8-b55b-93872c815747\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" Nov 02 20:33:18.229557 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.229522 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll"] Nov 02 20:33:18.242550 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.242515 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d9zn\" (UniqueName: \"kubernetes.io/projected/b90f9ab7-7575-4bd8-b55b-93872c815747-kube-api-access-9d9zn\") pod \"payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg\" (UID: \"b90f9ab7-7575-4bd8-b55b-93872c815747\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" Nov 02 20:33:18.289581 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.289479 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dl6n\" (UniqueName: \"kubernetes.io/projected/1797345c-e27c-423d-8b0c-71270e96647f-kube-api-access-9dl6n\") pod \"payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll\" (UID: \"1797345c-e27c-423d-8b0c-71270e96647f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" Nov 02 20:33:18.290086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.289640 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797345c-e27c-423d-8b0c-71270e96647f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll\" (UID: \"1797345c-e27c-423d-8b0c-71270e96647f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" Nov 02 20:33:18.341275 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.341241 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" Nov 02 20:33:18.394286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.391303 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dl6n\" (UniqueName: \"kubernetes.io/projected/1797345c-e27c-423d-8b0c-71270e96647f-kube-api-access-9dl6n\") pod \"payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll\" (UID: \"1797345c-e27c-423d-8b0c-71270e96647f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" Nov 02 20:33:18.394286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.392427 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797345c-e27c-423d-8b0c-71270e96647f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll\" (UID: \"1797345c-e27c-423d-8b0c-71270e96647f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" Nov 02 20:33:18.394286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.393553 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797345c-e27c-423d-8b0c-71270e96647f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll\" (UID: \"1797345c-e27c-423d-8b0c-71270e96647f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" Nov 02 20:33:18.446714 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.446677 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dl6n\" (UniqueName: \"kubernetes.io/projected/1797345c-e27c-423d-8b0c-71270e96647f-kube-api-access-9dl6n\") pod \"payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll\" (UID: \"1797345c-e27c-423d-8b0c-71270e96647f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" Nov 02 20:33:18.501521 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.501491 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" Nov 02 20:33:18.717966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.717931 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg"] Nov 02 20:33:18.724761 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:33:18.724718 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90f9ab7_7575_4bd8_b55b_93872c815747.slice/crio-da6c1d2ba1071f4b060277bed7d0e09dc78d6766a22e63262dde566aa82e18f1 WatchSource:0}: Error finding container da6c1d2ba1071f4b060277bed7d0e09dc78d6766a22e63262dde566aa82e18f1: Status 404 returned error can't find the container with id da6c1d2ba1071f4b060277bed7d0e09dc78d6766a22e63262dde566aa82e18f1 Nov 02 20:33:18.727980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.727957 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:18.876181 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.876097 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll"] Nov 02 20:33:18.883648 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:33:18.883620 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1797345c_e27c_423d_8b0c_71270e96647f.slice/crio-9b69929fff8b6631cd0b8b19ea1675238d5c016d497598d73d22a3a50c190d11 WatchSource:0}: Error finding container 9b69929fff8b6631cd0b8b19ea1675238d5c016d497598d73d22a3a50c190d11: Status 404 returned error can't find the container with id 9b69929fff8b6631cd0b8b19ea1675238d5c016d497598d73d22a3a50c190d11 Nov 02 20:33:18.913135 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.913105 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" event={"ID":"b90f9ab7-7575-4bd8-b55b-93872c815747","Type":"ContainerStarted","Data":"7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a"} Nov 02 20:33:18.913261 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.913165 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" event={"ID":"b90f9ab7-7575-4bd8-b55b-93872c815747","Type":"ContainerStarted","Data":"da6c1d2ba1071f4b060277bed7d0e09dc78d6766a22e63262dde566aa82e18f1"} Nov 02 20:33:18.913261 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.913196 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" Nov 02 20:33:18.914459 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:18.914435 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" event={"ID":"1797345c-e27c-423d-8b0c-71270e96647f","Type":"ContainerStarted","Data":"9b69929fff8b6631cd0b8b19ea1675238d5c016d497598d73d22a3a50c190d11"} Nov 02 20:33:19.026230 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:19.026168 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" podStartSLOduration=2.026153084 podStartE2EDuration="2.026153084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:19.024846862 +0000 UTC m=+2435.832631079" watchObservedRunningTime="2026-11-02 20:33:19.026153084 +0000 UTC m=+2435.833937300" Nov 02 20:33:19.921540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:19.921495 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" event={"ID":"1797345c-e27c-423d-8b0c-71270e96647f","Type":"ContainerStarted","Data":"23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de"} Nov 02 20:33:19.922161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:19.921582 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" Nov 02 20:33:20.032050 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:20.031997 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" podStartSLOduration=2.031982597 podStartE2EDuration="2.031982597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:20.031059546 +0000 UTC m=+2436.838843794" watchObservedRunningTime="2026-11-02 20:33:20.031982597 +0000 UTC m=+2436.839766813" Nov 02 20:33:20.362461 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:20.362364 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:20.668016 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:20.667981 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:20.839143 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:20.839104 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:21.284641 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:21.284603 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:21.852595 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:21.852566 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:22.293321 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:22.293285 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:22.902249 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:22.902217 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:23.085823 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:23.085784 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:24.634907 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:24.634800 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:24.807618 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:24.807587 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:24.839562 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:24.839516 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8"] Nov 02 20:33:24.839809 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:24.839768 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" podUID="657eca17-2790-4b44-9264-40048be97ab1" containerName="maas-api" containerID="cri-o://d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b" gracePeriod=30 Nov 02 20:33:24.978655 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:24.978613 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:25.126567 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.126533 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:33:25.179345 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.179308 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657eca17-2790-4b44-9264-40048be97ab1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/657eca17-2790-4b44-9264-40048be97ab1-openshift-service-ca\") pod \"657eca17-2790-4b44-9264-40048be97ab1\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " Nov 02 20:33:25.179540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.179404 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-metrics-certs\") pod \"657eca17-2790-4b44-9264-40048be97ab1\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " Nov 02 20:33:25.179615 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.179543 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657eca17-2790-4b44-9264-40048be97ab1-kube-api-access-kbtvg\" (UniqueName: \"kubernetes.io/projected/657eca17-2790-4b44-9264-40048be97ab1-kube-api-access-kbtvg\") pod \"657eca17-2790-4b44-9264-40048be97ab1\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " Nov 02 20:33:25.179615 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.179591 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls\") pod \"657eca17-2790-4b44-9264-40048be97ab1\" (UID: \"657eca17-2790-4b44-9264-40048be97ab1\") " Nov 02 20:33:25.179757 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.179728 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657eca17-2790-4b44-9264-40048be97ab1-openshift-service-ca" pod "657eca17-2790-4b44-9264-40048be97ab1" (UID: "657eca17-2790-4b44-9264-40048be97ab1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:25.180023 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.180001 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/657eca17-2790-4b44-9264-40048be97ab1-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:25.182246 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.182215 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-metrics-certs" pod "657eca17-2790-4b44-9264-40048be97ab1" (UID: "657eca17-2790-4b44-9264-40048be97ab1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:25.182361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.182240 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657eca17-2790-4b44-9264-40048be97ab1-kube-api-access-kbtvg" pod "657eca17-2790-4b44-9264-40048be97ab1" (UID: "657eca17-2790-4b44-9264-40048be97ab1"). InnerVolumeSpecName "kube-api-access-kbtvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:25.182361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.182288 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls" pod "657eca17-2790-4b44-9264-40048be97ab1" (UID: "657eca17-2790-4b44-9264-40048be97ab1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:25.281340 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.281244 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:25.281340 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.281277 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbtvg\" (UniqueName: \"kubernetes.io/projected/657eca17-2790-4b44-9264-40048be97ab1-kube-api-access-kbtvg\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:25.281340 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.281295 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/657eca17-2790-4b44-9264-40048be97ab1-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:25.959514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.959479 5561 generic.go:356] "Generic (PLEG): container finished" podID="657eca17-2790-4b44-9264-40048be97ab1" containerID="d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b" exitCode=0 Nov 02 20:33:25.959968 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.959533 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" Nov 02 20:33:25.959968 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.959540 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" event={"ID":"657eca17-2790-4b44-9264-40048be97ab1","Type":"ContainerDied","Data":"d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b"} Nov 02 20:33:25.959968 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.959596 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8" event={"ID":"657eca17-2790-4b44-9264-40048be97ab1","Type":"ContainerDied","Data":"407091213bbc444effee7babfc93d521e7543468510fc21d1958ab2e698f7dfb"} Nov 02 20:33:25.959968 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.959619 5561 scope.go:122] "RemoveContainer" containerID="d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b" Nov 02 20:33:25.969349 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.969327 5561 scope.go:122] "RemoveContainer" containerID="d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b" Nov 02 20:33:25.969638 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:33:25.969620 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b\": container with ID starting with d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b not found: ID does not exist" containerID="d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b" Nov 02 20:33:25.969725 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:25.969647 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b"} err="failed to get container status \"d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b\": rpc error: code = NotFound desc = could not find container \"d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b\": container with ID starting with d280e00785af08b59817285fd4ed0b9737eb7a076926be988819a169eabdf70b not found: ID does not exist" Nov 02 20:33:26.132709 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:26.132672 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8"] Nov 02 20:33:26.180037 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:26.180004 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1ab501-65dbf9c588-qrbf8"] Nov 02 20:33:26.509990 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:26.509950 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:26.778024 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:26.777933 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:26.956376 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:26.956346 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:27.313541 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:27.313502 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657eca17-2790-4b44-9264-40048be97ab1" path="/var/lib/kubelet/pods/657eca17-2790-4b44-9264-40048be97ab1/volumes" Nov 02 20:33:27.329171 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:27.329144 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:27.510654 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:27.510617 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:27.702232 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:27.702186 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:28.175355 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:28.175322 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:28.496454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:28.496372 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:28.679731 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:28.679692 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:29.924996 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:29.924965 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" Nov 02 20:33:30.929854 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:30.929822 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" Nov 02 20:33:31.747838 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:31.747790 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:31.931648 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:31.931613 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:32.126359 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.126278 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8"] Nov 02 20:33:32.126771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.126758 5561 cpu_manager.go:414] "containerMap: removing container" podUID="657eca17-2790-4b44-9264-40048be97ab1" containerName="maas-api" Nov 02 20:33:32.126817 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.126772 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657eca17-2790-4b44-9264-40048be97ab1" containerName="maas-api" Nov 02 20:33:32.126907 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.126867 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="657eca17-2790-4b44-9264-40048be97ab1" containerName="maas-api" Nov 02 20:33:32.131499 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.131478 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" Nov 02 20:33:32.159105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.159079 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p45h6\" (UniqueName: \"kubernetes.io/projected/39722652-3460-46ed-89ce-5ab8a3eb255d-kube-api-access-p45h6\") pod \"payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8\" (UID: \"39722652-3460-46ed-89ce-5ab8a3eb255d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" Nov 02 20:33:32.159255 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.159157 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39722652-3460-46ed-89ce-5ab8a3eb255d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8\" (UID: \"39722652-3460-46ed-89ce-5ab8a3eb255d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" Nov 02 20:33:32.183352 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.183323 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8"] Nov 02 20:33:32.225631 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.225597 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:32.228353 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.228334 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2b3b9e\"" Nov 02 20:33:32.228449 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.228365 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2b3b9e-dockercfg-z9wsr\"" Nov 02 20:33:32.260436 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.260401 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p45h6\" (UniqueName: \"kubernetes.io/projected/39722652-3460-46ed-89ce-5ab8a3eb255d-kube-api-access-p45h6\") pod \"payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8\" (UID: \"39722652-3460-46ed-89ce-5ab8a3eb255d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" Nov 02 20:33:32.260615 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.260512 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39722652-3460-46ed-89ce-5ab8a3eb255d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8\" (UID: \"39722652-3460-46ed-89ce-5ab8a3eb255d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" Nov 02 20:33:32.261234 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.261215 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39722652-3460-46ed-89ce-5ab8a3eb255d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8\" (UID: \"39722652-3460-46ed-89ce-5ab8a3eb255d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" Nov 02 20:33:32.278008 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.277981 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn"] Nov 02 20:33:32.282013 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.281990 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" Nov 02 20:33:32.347899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.330395 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p45h6\" (UniqueName: \"kubernetes.io/projected/39722652-3460-46ed-89ce-5ab8a3eb255d-kube-api-access-p45h6\") pod \"payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8\" (UID: \"39722652-3460-46ed-89ce-5ab8a3eb255d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" Nov 02 20:33:32.369904 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.349487 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn"] Nov 02 20:33:32.370199 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.362586 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfmj9\" (UniqueName: \"kubernetes.io/projected/cf0ee610-89f1-4e6f-9017-1d3777be0b29-kube-api-access-dfmj9\") pod \"payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn\" (UID: \"cf0ee610-89f1-4e6f-9017-1d3777be0b29\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" Nov 02 20:33:32.370651 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.370620 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0ee610-89f1-4e6f-9017-1d3777be0b29-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn\" (UID: \"cf0ee610-89f1-4e6f-9017-1d3777be0b29\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" Nov 02 20:33:32.444659 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.444625 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" Nov 02 20:33:32.471550 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.471518 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0ee610-89f1-4e6f-9017-1d3777be0b29-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn\" (UID: \"cf0ee610-89f1-4e6f-9017-1d3777be0b29\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" Nov 02 20:33:32.471699 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.471570 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfmj9\" (UniqueName: \"kubernetes.io/projected/cf0ee610-89f1-4e6f-9017-1d3777be0b29-kube-api-access-dfmj9\") pod \"payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn\" (UID: \"cf0ee610-89f1-4e6f-9017-1d3777be0b29\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" Nov 02 20:33:32.472275 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.472252 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0ee610-89f1-4e6f-9017-1d3777be0b29-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn\" (UID: \"cf0ee610-89f1-4e6f-9017-1d3777be0b29\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" Nov 02 20:33:32.525487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.525456 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfmj9\" (UniqueName: \"kubernetes.io/projected/cf0ee610-89f1-4e6f-9017-1d3777be0b29-kube-api-access-dfmj9\") pod \"payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn\" (UID: \"cf0ee610-89f1-4e6f-9017-1d3777be0b29\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" Nov 02 20:33:32.594999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.594965 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" Nov 02 20:33:32.816826 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.816755 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8"] Nov 02 20:33:32.823305 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:33:32.823274 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39722652_3460_46ed_89ce_5ab8a3eb255d.slice/crio-6c8346316cf317ab55d29625ad242d638f89eb9b6c20fd6bb23c492cc1f0d0d7 WatchSource:0}: Error finding container 6c8346316cf317ab55d29625ad242d638f89eb9b6c20fd6bb23c492cc1f0d0d7: Status 404 returned error can't find the container with id 6c8346316cf317ab55d29625ad242d638f89eb9b6c20fd6bb23c492cc1f0d0d7 Nov 02 20:33:32.968944 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:32.968914 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn"] Nov 02 20:33:32.975827 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:33:32.975800 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0ee610_89f1_4e6f_9017_1d3777be0b29.slice/crio-73ed0dd9c968bd886070c4c0f6be46d809414ad668f51fe024709520a7dfc0ef WatchSource:0}: Error finding container 73ed0dd9c968bd886070c4c0f6be46d809414ad668f51fe024709520a7dfc0ef: Status 404 returned error can't find the container with id 73ed0dd9c968bd886070c4c0f6be46d809414ad668f51fe024709520a7dfc0ef Nov 02 20:33:33.001718 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:33.001687 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" event={"ID":"39722652-3460-46ed-89ce-5ab8a3eb255d","Type":"ContainerStarted","Data":"d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40"} Nov 02 20:33:33.001834 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:33.001750 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" event={"ID":"39722652-3460-46ed-89ce-5ab8a3eb255d","Type":"ContainerStarted","Data":"6c8346316cf317ab55d29625ad242d638f89eb9b6c20fd6bb23c492cc1f0d0d7"} Nov 02 20:33:33.001914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:33.001897 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" Nov 02 20:33:33.002918 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:33.002897 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" event={"ID":"cf0ee610-89f1-4e6f-9017-1d3777be0b29","Type":"ContainerStarted","Data":"73ed0dd9c968bd886070c4c0f6be46d809414ad668f51fe024709520a7dfc0ef"} Nov 02 20:33:33.103538 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:33.103453 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b"] Nov 02 20:33:33.109721 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:33.109659 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" podStartSLOduration=1.109640935 podStartE2EDuration="1.109640935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:33.109072949 +0000 UTC m=+2449.916857166" watchObservedRunningTime="2026-11-02 20:33:33.109640935 +0000 UTC m=+2449.917425152" Nov 02 20:33:33.223973 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:33.223925 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:33.468642 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:33.468570 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:33.659601 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:33.659558 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:33.784762 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:33:33.784676 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:33:33.846234 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:33.846198 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:34.010313 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:34.010273 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" event={"ID":"cf0ee610-89f1-4e6f-9017-1d3777be0b29","Type":"ContainerStarted","Data":"8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4"} Nov 02 20:33:34.010740 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:34.010518 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" Nov 02 20:33:34.134385 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:34.134287 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" podStartSLOduration=2.134273616 podStartE2EDuration="2.134273616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:34.133739234 +0000 UTC m=+2450.941523451" watchObservedRunningTime="2026-11-02 20:33:34.134273616 +0000 UTC m=+2450.942057833" Nov 02 20:33:35.913303 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:35.913270 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj"] Nov 02 20:33:35.913769 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:35.913467 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" podUID="de79c7aa-d3aa-4209-adc4-53d5e54c5968" containerName="maas-api" containerID="cri-o://aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515" gracePeriod=30 Nov 02 20:33:36.090675 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.090626 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:36.231173 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.231133 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:33:36.320799 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.320760 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de79c7aa-d3aa-4209-adc4-53d5e54c5968-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de79c7aa-d3aa-4209-adc4-53d5e54c5968-openshift-service-ca\") pod \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " Nov 02 20:33:36.321011 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.320811 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-maas-api-tls\") pod \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " Nov 02 20:33:36.321011 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.320927 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-metrics-certs\") pod \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " Nov 02 20:33:36.321139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.321014 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de79c7aa-d3aa-4209-adc4-53d5e54c5968-kube-api-access-s5lq8\" (UniqueName: \"kubernetes.io/projected/de79c7aa-d3aa-4209-adc4-53d5e54c5968-kube-api-access-s5lq8\") pod \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\" (UID: \"de79c7aa-d3aa-4209-adc4-53d5e54c5968\") " Nov 02 20:33:36.321230 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.321202 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de79c7aa-d3aa-4209-adc4-53d5e54c5968-openshift-service-ca" pod "de79c7aa-d3aa-4209-adc4-53d5e54c5968" (UID: "de79c7aa-d3aa-4209-adc4-53d5e54c5968"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:36.321469 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.321448 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de79c7aa-d3aa-4209-adc4-53d5e54c5968-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:36.323531 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.323499 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-maas-api-tls" pod "de79c7aa-d3aa-4209-adc4-53d5e54c5968" (UID: "de79c7aa-d3aa-4209-adc4-53d5e54c5968"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:36.323728 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.323708 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-metrics-certs" pod "de79c7aa-d3aa-4209-adc4-53d5e54c5968" (UID: "de79c7aa-d3aa-4209-adc4-53d5e54c5968"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:36.323788 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.323749 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de79c7aa-d3aa-4209-adc4-53d5e54c5968-kube-api-access-s5lq8" pod "de79c7aa-d3aa-4209-adc4-53d5e54c5968" (UID: "de79c7aa-d3aa-4209-adc4-53d5e54c5968"). InnerVolumeSpecName "kube-api-access-s5lq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:36.423248 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.423011 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:36.423248 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.423037 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de79c7aa-d3aa-4209-adc4-53d5e54c5968-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:36.423248 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:36.423047 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5lq8\" (UniqueName: \"kubernetes.io/projected/de79c7aa-d3aa-4209-adc4-53d5e54c5968-kube-api-access-s5lq8\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:37.038299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:37.038262 5561 generic.go:356] "Generic (PLEG): container finished" podID="de79c7aa-d3aa-4209-adc4-53d5e54c5968" containerID="aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515" exitCode=0 Nov 02 20:33:37.038772 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:37.038313 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" Nov 02 20:33:37.038772 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:37.038333 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" event={"ID":"de79c7aa-d3aa-4209-adc4-53d5e54c5968","Type":"ContainerDied","Data":"aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515"} Nov 02 20:33:37.038772 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:37.038387 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj" event={"ID":"de79c7aa-d3aa-4209-adc4-53d5e54c5968","Type":"ContainerDied","Data":"354a52ab87693881257c45c64b0f7a1aefc9677a5e3905687e41a6541aa40467"} Nov 02 20:33:37.038772 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:37.038423 5561 scope.go:122] "RemoveContainer" containerID="aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515" Nov 02 20:33:37.049048 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:37.049027 5561 scope.go:122] "RemoveContainer" containerID="aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515" Nov 02 20:33:37.049292 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:33:37.049272 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515\": container with ID starting with aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515 not found: ID does not exist" containerID="aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515" Nov 02 20:33:37.049341 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:37.049302 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515"} err="failed to get container status \"aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515\": rpc error: code = NotFound desc = could not find container \"aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515\": container with ID starting with aa7fcea97fb86ef321e7ed0856e9ccf226b2c777c379e60c6ac4fde585ff9515 not found: ID does not exist" Nov 02 20:33:37.193388 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:37.193353 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj"] Nov 02 20:33:37.197307 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:37.197282 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9fcedd-5db45f855b-9blkj"] Nov 02 20:33:37.313637 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:37.313557 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de79c7aa-d3aa-4209-adc4-53d5e54c5968" path="/var/lib/kubelet/pods/de79c7aa-d3aa-4209-adc4-53d5e54c5968/volumes" Nov 02 20:33:37.592063 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:37.591971 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:37.890894 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:37.890794 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:38.059350 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:38.059316 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:38.518235 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:38.518200 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:41.282015 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:41.281977 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:41.447653 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:41.447516 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:42.875545 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:42.875508 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:44.013353 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:44.013324 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" Nov 02 20:33:44.339559 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:44.339449 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:44.630256 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:44.630141 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:44.645157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:44.645127 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5"] Nov 02 20:33:44.645361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:44.645339 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerName="istio-proxy" containerID="cri-o://7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786" gracePeriod=30 Nov 02 20:33:44.808784 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:44.808748 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:45.020294 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:45.020264 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" Nov 02 20:33:47.063785 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.063749 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl"] Nov 02 20:33:47.064263 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.064026 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" podUID="de0de27e-70b6-4f5d-b269-2e1651c6dd53" containerName="maas-api" containerID="cri-o://b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9" gracePeriod=30 Nov 02 20:33:47.205701 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.205645 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:47.361416 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.361388 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:33:47.458307 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.458279 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de0de27e-70b6-4f5d-b269-2e1651c6dd53-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0de27e-70b6-4f5d-b269-2e1651c6dd53-openshift-service-ca\") pod \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " Nov 02 20:33:47.458520 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.458501 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0de27e-70b6-4f5d-b269-2e1651c6dd53-kube-api-access-dw6q8\" (UniqueName: \"kubernetes.io/projected/de0de27e-70b6-4f5d-b269-2e1651c6dd53-kube-api-access-dw6q8\") pod \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " Nov 02 20:33:47.458667 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.458648 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-metrics-certs\") pod \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " Nov 02 20:33:47.458722 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.458671 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0de27e-70b6-4f5d-b269-2e1651c6dd53-openshift-service-ca" pod "de0de27e-70b6-4f5d-b269-2e1651c6dd53" (UID: "de0de27e-70b6-4f5d-b269-2e1651c6dd53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:47.458789 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.458732 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-maas-api-tls\") pod \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\" (UID: \"de0de27e-70b6-4f5d-b269-2e1651c6dd53\") " Nov 02 20:33:47.460302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.460276 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de0de27e-70b6-4f5d-b269-2e1651c6dd53-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:47.460968 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.460939 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-metrics-certs" pod "de0de27e-70b6-4f5d-b269-2e1651c6dd53" (UID: "de0de27e-70b6-4f5d-b269-2e1651c6dd53"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:47.461264 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.461243 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-maas-api-tls" pod "de0de27e-70b6-4f5d-b269-2e1651c6dd53" (UID: "de0de27e-70b6-4f5d-b269-2e1651c6dd53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:47.461325 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.461263 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0de27e-70b6-4f5d-b269-2e1651c6dd53-kube-api-access-dw6q8" pod "de0de27e-70b6-4f5d-b269-2e1651c6dd53" (UID: "de0de27e-70b6-4f5d-b269-2e1651c6dd53"). InnerVolumeSpecName "kube-api-access-dw6q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:47.562048 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.562012 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw6q8\" (UniqueName: \"kubernetes.io/projected/de0de27e-70b6-4f5d-b269-2e1651c6dd53-kube-api-access-dw6q8\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:47.562048 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.562045 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:47.562048 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.562056 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de0de27e-70b6-4f5d-b269-2e1651c6dd53-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:47.893781 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:47.893715 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:48.065297 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.065255 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:48.102302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.102267 5561 generic.go:356] "Generic (PLEG): container finished" podID="de0de27e-70b6-4f5d-b269-2e1651c6dd53" containerID="b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9" exitCode=0 Nov 02 20:33:48.102514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.102316 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" Nov 02 20:33:48.102514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.102323 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" event={"ID":"de0de27e-70b6-4f5d-b269-2e1651c6dd53","Type":"ContainerDied","Data":"b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9"} Nov 02 20:33:48.102514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.102376 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl" event={"ID":"de0de27e-70b6-4f5d-b269-2e1651c6dd53","Type":"ContainerDied","Data":"5b338fc625b9ab753e164a1506629b13f90e311531eea0cc414476af03dffc2c"} Nov 02 20:33:48.102514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.102404 5561 scope.go:122] "RemoveContainer" containerID="b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9" Nov 02 20:33:48.112579 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.112563 5561 scope.go:122] "RemoveContainer" containerID="b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9" Nov 02 20:33:48.112858 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:33:48.112840 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9\": container with ID starting with b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9 not found: ID does not exist" containerID="b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9" Nov 02 20:33:48.112920 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.112866 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9"} err="failed to get container status \"b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9\": rpc error: code = NotFound desc = could not find container \"b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9\": container with ID starting with b9710bb5dfd5ca83b920fcb3594d204f1c6549b4607d0e46dfc5ce8c61afa0c9 not found: ID does not exist" Nov 02 20:33:48.124815 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.124786 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" containerID="cri-o://22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5" gracePeriod=45 Nov 02 20:33:48.205316 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.205276 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": dial tcp 10.134.0.43:8000: connect: connection refused" Nov 02 20:33:48.258035 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.257972 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl"] Nov 02 20:33:48.264900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.264583 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62d287-747fccc78c-s6hbl"] Nov 02 20:33:48.393474 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.393449 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:33:48.471708 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.471615 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-model-cache\") pod \"8e29f5a7-5aa5-441e-befd-42f50aefefde\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " Nov 02 20:33:48.471708 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.471659 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-home\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-home\") pod \"8e29f5a7-5aa5-441e-befd-42f50aefefde\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " Nov 02 20:33:48.471708 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.471692 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e29f5a7-5aa5-441e-befd-42f50aefefde-kube-api-access-c5272\" (UniqueName: \"kubernetes.io/projected/8e29f5a7-5aa5-441e-befd-42f50aefefde-kube-api-access-c5272\") pod \"8e29f5a7-5aa5-441e-befd-42f50aefefde\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " Nov 02 20:33:48.472043 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.471864 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-tmp-dir\") pod \"8e29f5a7-5aa5-441e-befd-42f50aefefde\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " Nov 02 20:33:48.472043 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.471924 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-model-cache" pod "8e29f5a7-5aa5-441e-befd-42f50aefefde" (UID: "8e29f5a7-5aa5-441e-befd-42f50aefefde"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:48.472043 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.471947 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-home" pod "8e29f5a7-5aa5-441e-befd-42f50aefefde" (UID: "8e29f5a7-5aa5-441e-befd-42f50aefefde"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:48.472043 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.471982 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e29f5a7-5aa5-441e-befd-42f50aefefde-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e29f5a7-5aa5-441e-befd-42f50aefefde-tls-certs\") pod \"8e29f5a7-5aa5-441e-befd-42f50aefefde\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " Nov 02 20:33:48.472043 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.472037 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-dshm\") pod \"8e29f5a7-5aa5-441e-befd-42f50aefefde\" (UID: \"8e29f5a7-5aa5-441e-befd-42f50aefefde\") " Nov 02 20:33:48.472280 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.472098 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-tmp-dir" pod "8e29f5a7-5aa5-441e-befd-42f50aefefde" (UID: "8e29f5a7-5aa5-441e-befd-42f50aefefde"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:48.472439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.472422 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.472510 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.472443 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.472510 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.472457 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.474406 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.474378 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e29f5a7-5aa5-441e-befd-42f50aefefde-tls-certs" pod "8e29f5a7-5aa5-441e-befd-42f50aefefde" (UID: "8e29f5a7-5aa5-441e-befd-42f50aefefde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:48.474538 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.474504 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-dshm" pod "8e29f5a7-5aa5-441e-befd-42f50aefefde" (UID: "8e29f5a7-5aa5-441e-befd-42f50aefefde"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:48.474538 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.474529 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e29f5a7-5aa5-441e-befd-42f50aefefde-kube-api-access-c5272" pod "8e29f5a7-5aa5-441e-befd-42f50aefefde" (UID: "8e29f5a7-5aa5-441e-befd-42f50aefefde"). InnerVolumeSpecName "kube-api-access-c5272". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:48.573469 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.573435 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5272\" (UniqueName: \"kubernetes.io/projected/8e29f5a7-5aa5-441e-befd-42f50aefefde-kube-api-access-c5272\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.573469 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.573465 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e29f5a7-5aa5-441e-befd-42f50aefefde-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.573469 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.573475 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e29f5a7-5aa5-441e-befd-42f50aefefde-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.730068 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:48.729979 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:49.006181 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.006103 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:49.110863 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.110822 5561 generic.go:356] "Generic (PLEG): container finished" podID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerID="22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5" exitCode=0 Nov 02 20:33:49.112615 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.111065 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" Nov 02 20:33:49.112615 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.111910 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" event={"ID":"8e29f5a7-5aa5-441e-befd-42f50aefefde","Type":"ContainerDied","Data":"22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5"} Nov 02 20:33:49.112615 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.111957 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b" event={"ID":"8e29f5a7-5aa5-441e-befd-42f50aefefde","Type":"ContainerDied","Data":"d8cff6b313d9c5b61b2c4461c6327f8b95418bac4444c9b521b6c39ac65913da"} Nov 02 20:33:49.112615 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.111982 5561 scope.go:122] "RemoveContainer" containerID="22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5" Nov 02 20:33:49.132628 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.132605 5561 scope.go:122] "RemoveContainer" containerID="22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5" Nov 02 20:33:49.133551 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:33:49.133519 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5\": container with ID starting with 22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5 not found: ID does not exist" containerID="22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5" Nov 02 20:33:49.134454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.133563 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5"} err="failed to get container status \"22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5\": rpc error: code = NotFound desc = could not find container \"22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5\": container with ID starting with 22e4c0de24366ca9943414a04f7bbcaca92edf5a845777a32b02435bf21a9ff5 not found: ID does not exist" Nov 02 20:33:49.184869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.184835 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:49.270521 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.270438 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b"] Nov 02 20:33:49.272276 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.272253 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/simulator-w1-accdf5-kserve-67d8987bb6-mss6b"] Nov 02 20:33:49.313277 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.313243 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" path="/var/lib/kubelet/pods/8e29f5a7-5aa5-441e-befd-42f50aefefde/volumes" Nov 02 20:33:49.314177 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.314150 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de0de27e-70b6-4f5d-b269-2e1651c6dd53" path="/var/lib/kubelet/pods/de0de27e-70b6-4f5d-b269-2e1651c6dd53/volumes" Nov 02 20:33:49.604388 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.604299 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:49.900571 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.900548 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:33:49.989247 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.989214 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-kube-api-access-mz8qd\" (UniqueName: \"kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-kube-api-access-mz8qd\") pod \"cba153ce-9124-483e-b2d4-ae582cca5f6b\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " Nov 02 20:33:49.989425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.989277 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-podinfo\") pod \"cba153ce-9124-483e-b2d4-ae582cca5f6b\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " Nov 02 20:33:49.989425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.989315 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-credential-socket\") pod \"cba153ce-9124-483e-b2d4-ae582cca5f6b\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " Nov 02 20:33:49.989425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.989345 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-data\") pod \"cba153ce-9124-483e-b2d4-ae582cca5f6b\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " Nov 02 20:33:49.989425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.989407 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-socket\") pod \"cba153ce-9124-483e-b2d4-ae582cca5f6b\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " Nov 02 20:33:49.989650 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.989446 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba153ce-9124-483e-b2d4-ae582cca5f6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba153ce-9124-483e-b2d4-ae582cca5f6b-istiod-ca-cert\") pod \"cba153ce-9124-483e-b2d4-ae582cca5f6b\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " Nov 02 20:33:49.989650 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.989484 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-envoy\") pod \"cba153ce-9124-483e-b2d4-ae582cca5f6b\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " Nov 02 20:33:49.989650 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.989521 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-token\" (UniqueName: \"kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-token\") pod \"cba153ce-9124-483e-b2d4-ae582cca5f6b\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " Nov 02 20:33:49.989650 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.989592 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-certs\") pod \"cba153ce-9124-483e-b2d4-ae582cca5f6b\" (UID: \"cba153ce-9124-483e-b2d4-ae582cca5f6b\") " Nov 02 20:33:49.989869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.989648 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-data" pod "cba153ce-9124-483e-b2d4-ae582cca5f6b" (UID: "cba153ce-9124-483e-b2d4-ae582cca5f6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:49.989869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.989660 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-credential-socket" pod "cba153ce-9124-483e-b2d4-ae582cca5f6b" (UID: "cba153ce-9124-483e-b2d4-ae582cca5f6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:49.990028 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.990002 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba153ce-9124-483e-b2d4-ae582cca5f6b-istiod-ca-cert" pod "cba153ce-9124-483e-b2d4-ae582cca5f6b" (UID: "cba153ce-9124-483e-b2d4-ae582cca5f6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:49.990126 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.990109 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba153ce-9124-483e-b2d4-ae582cca5f6b-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:49.990209 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.990185 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-certs" pod "cba153ce-9124-483e-b2d4-ae582cca5f6b" (UID: "cba153ce-9124-483e-b2d4-ae582cca5f6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:49.990209 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.990203 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:49.990318 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.990216 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:49.990656 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.990061 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-socket" pod "cba153ce-9124-483e-b2d4-ae582cca5f6b" (UID: "cba153ce-9124-483e-b2d4-ae582cca5f6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:49.991635 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.991614 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-podinfo" pod "cba153ce-9124-483e-b2d4-ae582cca5f6b" (UID: "cba153ce-9124-483e-b2d4-ae582cca5f6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:33:49.991937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.991915 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-token" pod "cba153ce-9124-483e-b2d4-ae582cca5f6b" (UID: "cba153ce-9124-483e-b2d4-ae582cca5f6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:49.992007 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.991992 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-envoy" pod "cba153ce-9124-483e-b2d4-ae582cca5f6b" (UID: "cba153ce-9124-483e-b2d4-ae582cca5f6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:49.992050 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:49.992016 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-kube-api-access-mz8qd" pod "cba153ce-9124-483e-b2d4-ae582cca5f6b" (UID: "cba153ce-9124-483e-b2d4-ae582cca5f6b"). InnerVolumeSpecName "kube-api-access-mz8qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:50.091091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.091054 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.091091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.091087 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.091091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.091098 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz8qd\" (UniqueName: \"kubernetes.io/projected/cba153ce-9124-483e-b2d4-ae582cca5f6b-kube-api-access-mz8qd\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.091389 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.091108 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.091389 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.091117 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.091389 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.091125 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba153ce-9124-483e-b2d4-ae582cca5f6b-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.122288 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.122257 5561 generic.go:356] "Generic (PLEG): container finished" podID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerID="7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786" exitCode=0 Nov 02 20:33:50.122716 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.122299 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" Nov 02 20:33:50.122716 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.122349 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" event={"ID":"cba153ce-9124-483e-b2d4-ae582cca5f6b","Type":"ContainerDied","Data":"7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786"} Nov 02 20:33:50.122716 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.122400 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5" event={"ID":"cba153ce-9124-483e-b2d4-ae582cca5f6b","Type":"ContainerDied","Data":"7cb8ad37634a4c7fe156d86c6cf24fd42b6325cc679be4ed31117473ecd5b089"} Nov 02 20:33:50.122716 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.122425 5561 scope.go:122] "RemoveContainer" containerID="7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786" Nov 02 20:33:50.132267 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.132248 5561 scope.go:122] "RemoveContainer" containerID="7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786" Nov 02 20:33:50.132559 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:33:50.132537 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786\": container with ID starting with 7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786 not found: ID does not exist" containerID="7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786" Nov 02 20:33:50.132658 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.132567 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786"} err="failed to get container status \"7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786\": rpc error: code = NotFound desc = could not find container \"7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786\": container with ID starting with 7d4163cd3dd56f33462759ee8537f3bdeaa6ef0690c18769d497ee6c8d699786 not found: ID does not exist" Nov 02 20:33:50.278476 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.278440 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5"] Nov 02 20:33:50.281981 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:50.281956 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-mn8l5"] Nov 02 20:33:51.312614 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:51.312583 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" path="/var/lib/kubelet/pods/cba153ce-9124-483e-b2d4-ae582cca5f6b/volumes" Nov 02 20:33:53.922574 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:53.922534 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:54.990678 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:54.990640 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:55.454174 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:55.454139 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:55.711380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:55.711287 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:55.881668 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:55.881634 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:56.063941 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:56.063850 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:56.274106 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:56.274047 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:58.694765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:33:58.694713 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:03.455727 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:03.455680 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:03.780047 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:34:03.779966 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:34:06.612337 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:06.612302 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:08.312492 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:08.312454 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:10.296206 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:10.296156 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:10.523380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:10.523345 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:10.996847 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:10.996795 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:12.633903 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:12.633832 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:12.813777 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:12.813741 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:13.110012 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:13.109967 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:13.295361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:13.295303 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:13.475828 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:13.475769 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:16.543545 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:16.543512 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:16.721555 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:16.721524 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:17.866829 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:17.866775 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:22.290354 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:22.290312 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:26.694606 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:26.694564 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:26.867985 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:26.867946 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:31.314465 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:31.314425 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:33.780303 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:34:33.780264 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:34:35.781718 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:35.781658 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:39.919656 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.919617 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv"] Nov 02 20:34:39.920114 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.920099 5561 cpu_manager.go:414] "containerMap: removing container" podUID="de0de27e-70b6-4f5d-b269-2e1651c6dd53" containerName="maas-api" Nov 02 20:34:39.920158 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.920115 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0de27e-70b6-4f5d-b269-2e1651c6dd53" containerName="maas-api" Nov 02 20:34:39.920158 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.920144 5561 cpu_manager.go:414] "containerMap: removing container" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" Nov 02 20:34:39.920158 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.920150 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" Nov 02 20:34:39.920256 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.920172 5561 cpu_manager.go:414] "containerMap: removing container" podUID="de79c7aa-d3aa-4209-adc4-53d5e54c5968" containerName="maas-api" Nov 02 20:34:39.920256 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.920177 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de79c7aa-d3aa-4209-adc4-53d5e54c5968" containerName="maas-api" Nov 02 20:34:39.920256 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.920186 5561 cpu_manager.go:414] "containerMap: removing container" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerName="istio-proxy" Nov 02 20:34:39.920256 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.920190 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerName="istio-proxy" Nov 02 20:34:39.920375 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.920270 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="de79c7aa-d3aa-4209-adc4-53d5e54c5968" containerName="maas-api" Nov 02 20:34:39.920375 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.920279 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba153ce-9124-483e-b2d4-ae582cca5f6b" containerName="istio-proxy" Nov 02 20:34:39.920375 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.920286 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0de27e-70b6-4f5d-b269-2e1651c6dd53" containerName="maas-api" Nov 02 20:34:39.920375 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.920296 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e29f5a7-5aa5-441e-befd-42f50aefefde" containerName="main" Nov 02 20:34:39.923602 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:39.923575 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.020934 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.020859 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2b3b9e-serving-cert\"" Nov 02 20:34:40.022859 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.022834 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv"] Nov 02 20:34:40.054631 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.054593 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-maas-api-tls\") pod \"maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.054810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.054732 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-metrics-certs\") pod \"maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.054898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.054857 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq68q\" (UniqueName: \"kubernetes.io/projected/0db8b320-9073-44fd-b41e-cb96b8e76fcf-kube-api-access-dq68q\") pod \"maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.055151 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.055128 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db8b320-9073-44fd-b41e-cb96b8e76fcf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.147929 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.147835 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:40.155931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.155862 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-maas-api-tls\") pod \"maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.156385 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.156245 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-metrics-certs\") pod \"maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.159570 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.159540 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-metrics-certs\") pod \"maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.160223 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.160202 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-maas-api-tls\") pod \"maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.167267 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.166784 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq68q\" (UniqueName: \"kubernetes.io/projected/0db8b320-9073-44fd-b41e-cb96b8e76fcf-kube-api-access-dq68q\") pod \"maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.167757 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.167730 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db8b320-9073-44fd-b41e-cb96b8e76fcf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.168775 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.168646 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db8b320-9073-44fd-b41e-cb96b8e76fcf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.221010 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.220941 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq68q\" (UniqueName: \"kubernetes.io/projected/0db8b320-9073-44fd-b41e-cb96b8e76fcf-kube-api-access-dq68q\") pod \"maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.237358 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.237325 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:40.337190 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.334870 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:40.510036 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.509961 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:40.610435 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:40.610400 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv"] Nov 02 20:34:40.617766 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:34:40.617733 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db8b320_9073_44fd_b41e_cb96b8e76fcf.slice/crio-4324772e00cb8a29cab39034c1ac273acb457d40bc8a9d627bbe7f17cc4557bf WatchSource:0}: Error finding container 4324772e00cb8a29cab39034c1ac273acb457d40bc8a9d627bbe7f17cc4557bf: Status 404 returned error can't find the container with id 4324772e00cb8a29cab39034c1ac273acb457d40bc8a9d627bbe7f17cc4557bf Nov 02 20:34:41.405656 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:41.405620 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" event={"ID":"0db8b320-9073-44fd-b41e-cb96b8e76fcf","Type":"ContainerStarted","Data":"4324772e00cb8a29cab39034c1ac273acb457d40bc8a9d627bbe7f17cc4557bf"} Nov 02 20:34:42.427657 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:42.427613 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" event={"ID":"0db8b320-9073-44fd-b41e-cb96b8e76fcf","Type":"ContainerStarted","Data":"b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae"} Nov 02 20:34:42.428130 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:42.427805 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:42.537495 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:42.537439 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" podStartSLOduration=2.707483934 podStartE2EDuration="3.537425563s" totalImagesPullingTime="829.941629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:42.53571664 +0000 UTC m=+2519.343500874" watchObservedRunningTime="2026-11-02 20:34:42.537425563 +0000 UTC m=+2519.345209787" Nov 02 20:34:45.140488 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:45.140438 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:48.439735 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:48.439701 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:34:49.772563 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:49.772516 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:53.681026 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:53.680987 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:55.138852 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:55.138816 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:55.342086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:55.342053 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:55.519815 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:55.519778 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:56.094544 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:56.094512 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:58.932618 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:34:58.932583 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:00.159030 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:00.158996 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:01.862040 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:01.861999 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:02.039845 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:02.039808 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:02.214906 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:02.213945 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:03.779550 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:35:03.779514 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:35:04.794625 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:04.794586 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:09.006635 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:09.006605 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:10.357659 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:10.357621 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk"] Nov 02 20:35:10.358170 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:10.357841 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" podUID="a9874e15-3cc4-4b17-962a-94bf2d026d94" containerName="istio-proxy" containerID="cri-o://f104acf0f5f2761d809fb129a1a2220a421de38e4bea3a661c06e57e530f50b8" gracePeriod=30 Nov 02 20:35:10.457148 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:10.457106 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz"] Nov 02 20:35:10.457375 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:10.457351 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" podUID="c088f7da-23e3-4952-bb72-a9e91456d9a3" containerName="istio-proxy" containerID="cri-o://1142ad4f3a677345bbb04cb24fbef6ab670bfcff393f9e56fa61c069e00f4568" gracePeriod=30 Nov 02 20:35:10.697269 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:10.697208 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:11.348894 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:11.348841 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:11.628552 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:11.628447 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:13.733163 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:13.733118 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:13.992505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:13.992408 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:15.617276 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.617245 5561 generic.go:356] "Generic (PLEG): container finished" podID="c088f7da-23e3-4952-bb72-a9e91456d9a3" containerID="1142ad4f3a677345bbb04cb24fbef6ab670bfcff393f9e56fa61c069e00f4568" exitCode=0 Nov 02 20:35:15.617652 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.617312 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" event={"ID":"c088f7da-23e3-4952-bb72-a9e91456d9a3","Type":"ContainerDied","Data":"1142ad4f3a677345bbb04cb24fbef6ab670bfcff393f9e56fa61c069e00f4568"} Nov 02 20:35:15.618797 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.618772 5561 generic.go:356] "Generic (PLEG): container finished" podID="a9874e15-3cc4-4b17-962a-94bf2d026d94" containerID="f104acf0f5f2761d809fb129a1a2220a421de38e4bea3a661c06e57e530f50b8" exitCode=0 Nov 02 20:35:15.618928 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.618806 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" event={"ID":"a9874e15-3cc4-4b17-962a-94bf2d026d94","Type":"ContainerDied","Data":"f104acf0f5f2761d809fb129a1a2220a421de38e4bea3a661c06e57e530f50b8"} Nov 02 20:35:15.618928 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.618836 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" event={"ID":"a9874e15-3cc4-4b17-962a-94bf2d026d94","Type":"ContainerDied","Data":"9cc252836967006764ab30160ab7800cc3d9b9421ace40285805ef938fec8061"} Nov 02 20:35:15.618928 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.618854 5561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cc252836967006764ab30160ab7800cc3d9b9421ace40285805ef938fec8061" Nov 02 20:35:15.625087 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.625037 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:35:15.689519 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.689487 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-certs\") pod \"a9874e15-3cc4-4b17-962a-94bf2d026d94\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " Nov 02 20:35:15.689698 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.689528 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-kube-api-access-r8gzq\" (UniqueName: \"kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-kube-api-access-r8gzq\") pod \"a9874e15-3cc4-4b17-962a-94bf2d026d94\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " Nov 02 20:35:15.689698 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.689566 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-envoy\") pod \"a9874e15-3cc4-4b17-962a-94bf2d026d94\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " Nov 02 20:35:15.689698 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.689663 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-credential-socket\") pod \"a9874e15-3cc4-4b17-962a-94bf2d026d94\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " Nov 02 20:35:15.689698 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.689681 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-token\" (UniqueName: \"kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-token\") pod \"a9874e15-3cc4-4b17-962a-94bf2d026d94\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " Nov 02 20:35:15.689862 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.689701 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-data\") pod \"a9874e15-3cc4-4b17-962a-94bf2d026d94\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " Nov 02 20:35:15.689862 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.689741 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-socket\") pod \"a9874e15-3cc4-4b17-962a-94bf2d026d94\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " Nov 02 20:35:15.689862 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.689805 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-certs" pod "a9874e15-3cc4-4b17-962a-94bf2d026d94" (UID: "a9874e15-3cc4-4b17-962a-94bf2d026d94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.689862 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.689861 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-podinfo\") pod \"a9874e15-3cc4-4b17-962a-94bf2d026d94\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " Nov 02 20:35:15.690085 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.689920 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9874e15-3cc4-4b17-962a-94bf2d026d94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9874e15-3cc4-4b17-962a-94bf2d026d94-istiod-ca-cert\") pod \"a9874e15-3cc4-4b17-962a-94bf2d026d94\" (UID: \"a9874e15-3cc4-4b17-962a-94bf2d026d94\") " Nov 02 20:35:15.690146 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.690103 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-socket" pod "a9874e15-3cc4-4b17-962a-94bf2d026d94" (UID: "a9874e15-3cc4-4b17-962a-94bf2d026d94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.690397 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.690343 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-credential-socket" pod "a9874e15-3cc4-4b17-962a-94bf2d026d94" (UID: "a9874e15-3cc4-4b17-962a-94bf2d026d94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.690397 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.690360 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-data" pod "a9874e15-3cc4-4b17-962a-94bf2d026d94" (UID: "a9874e15-3cc4-4b17-962a-94bf2d026d94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.690581 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.690415 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9874e15-3cc4-4b17-962a-94bf2d026d94-istiod-ca-cert" pod "a9874e15-3cc4-4b17-962a-94bf2d026d94" (UID: "a9874e15-3cc4-4b17-962a-94bf2d026d94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:15.690581 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.690482 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.690581 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.690503 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.690581 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.690518 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.690581 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.690532 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.692980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.692939 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-podinfo" pod "a9874e15-3cc4-4b17-962a-94bf2d026d94" (UID: "a9874e15-3cc4-4b17-962a-94bf2d026d94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:35:15.692980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.692954 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-token" pod "a9874e15-3cc4-4b17-962a-94bf2d026d94" (UID: "a9874e15-3cc4-4b17-962a-94bf2d026d94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:15.692980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.692947 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-envoy" pod "a9874e15-3cc4-4b17-962a-94bf2d026d94" (UID: "a9874e15-3cc4-4b17-962a-94bf2d026d94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.693154 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.693038 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-kube-api-access-r8gzq" pod "a9874e15-3cc4-4b17-962a-94bf2d026d94" (UID: "a9874e15-3cc4-4b17-962a-94bf2d026d94"). InnerVolumeSpecName "kube-api-access-r8gzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:15.717742 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.717719 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:35:15.737373 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.737336 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:15.791120 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791086 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-envoy\") pod \"c088f7da-23e3-4952-bb72-a9e91456d9a3\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " Nov 02 20:35:15.791303 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791139 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-certs\") pod \"c088f7da-23e3-4952-bb72-a9e91456d9a3\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " Nov 02 20:35:15.791303 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791171 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-podinfo\") pod \"c088f7da-23e3-4952-bb72-a9e91456d9a3\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " Nov 02 20:35:15.791303 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791217 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-data\") pod \"c088f7da-23e3-4952-bb72-a9e91456d9a3\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " Nov 02 20:35:15.791303 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791273 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-kube-api-access-q8486\" (UniqueName: \"kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-kube-api-access-q8486\") pod \"c088f7da-23e3-4952-bb72-a9e91456d9a3\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " Nov 02 20:35:15.791505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791320 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-token\" (UniqueName: \"kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-token\") pod \"c088f7da-23e3-4952-bb72-a9e91456d9a3\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " Nov 02 20:35:15.791505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791364 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-credential-socket\") pod \"c088f7da-23e3-4952-bb72-a9e91456d9a3\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " Nov 02 20:35:15.791505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791420 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-socket\") pod \"c088f7da-23e3-4952-bb72-a9e91456d9a3\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " Nov 02 20:35:15.791505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791431 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-certs" pod "c088f7da-23e3-4952-bb72-a9e91456d9a3" (UID: "c088f7da-23e3-4952-bb72-a9e91456d9a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.791505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791446 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c088f7da-23e3-4952-bb72-a9e91456d9a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c088f7da-23e3-4952-bb72-a9e91456d9a3-istiod-ca-cert\") pod \"c088f7da-23e3-4952-bb72-a9e91456d9a3\" (UID: \"c088f7da-23e3-4952-bb72-a9e91456d9a3\") " Nov 02 20:35:15.791774 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791642 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-data" pod "c088f7da-23e3-4952-bb72-a9e91456d9a3" (UID: "c088f7da-23e3-4952-bb72-a9e91456d9a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.791774 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791663 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-credential-socket" pod "c088f7da-23e3-4952-bb72-a9e91456d9a3" (UID: "c088f7da-23e3-4952-bb72-a9e91456d9a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.791920 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791891 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-socket" pod "c088f7da-23e3-4952-bb72-a9e91456d9a3" (UID: "c088f7da-23e3-4952-bb72-a9e91456d9a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.792015 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791979 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.792015 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791986 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c088f7da-23e3-4952-bb72-a9e91456d9a3-istiod-ca-cert" pod "c088f7da-23e3-4952-bb72-a9e91456d9a3" (UID: "c088f7da-23e3-4952-bb72-a9e91456d9a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:15.792015 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.791999 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.792182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.792017 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.792182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.792032 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.792182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.792046 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.792182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.792059 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9874e15-3cc4-4b17-962a-94bf2d026d94-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.792182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.792074 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8gzq\" (UniqueName: \"kubernetes.io/projected/a9874e15-3cc4-4b17-962a-94bf2d026d94-kube-api-access-r8gzq\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.792182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.792090 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9874e15-3cc4-4b17-962a-94bf2d026d94-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.793629 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.793605 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-envoy" pod "c088f7da-23e3-4952-bb72-a9e91456d9a3" (UID: "c088f7da-23e3-4952-bb72-a9e91456d9a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.793732 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.793700 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-podinfo" pod "c088f7da-23e3-4952-bb72-a9e91456d9a3" (UID: "c088f7da-23e3-4952-bb72-a9e91456d9a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:35:15.794094 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.794071 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-kube-api-access-q8486" pod "c088f7da-23e3-4952-bb72-a9e91456d9a3" (UID: "c088f7da-23e3-4952-bb72-a9e91456d9a3"). InnerVolumeSpecName "kube-api-access-q8486". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:15.794242 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.794225 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-token" pod "c088f7da-23e3-4952-bb72-a9e91456d9a3" (UID: "c088f7da-23e3-4952-bb72-a9e91456d9a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:15.893447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.893359 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.893447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.893391 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.893447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.893401 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c088f7da-23e3-4952-bb72-a9e91456d9a3-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.893447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.893414 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.893447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.893428 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c088f7da-23e3-4952-bb72-a9e91456d9a3-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.893447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:15.893441 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8486\" (UniqueName: \"kubernetes.io/projected/c088f7da-23e3-4952-bb72-a9e91456d9a3-kube-api-access-q8486\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.626680 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:16.626645 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk" Nov 02 20:35:16.627181 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:16.626645 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" Nov 02 20:35:16.627181 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:16.626647 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz" event={"ID":"c088f7da-23e3-4952-bb72-a9e91456d9a3","Type":"ContainerDied","Data":"658191d851cc398a111f760fe1c4378197337f2693a12e23cf6f8dca8fe9f3e1"} Nov 02 20:35:16.627181 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:16.626808 5561 scope.go:122] "RemoveContainer" containerID="1142ad4f3a677345bbb04cb24fbef6ab670bfcff393f9e56fa61c069e00f4568" Nov 02 20:35:16.781737 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:16.781692 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk"] Nov 02 20:35:16.829255 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:16.829214 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-lztwk"] Nov 02 20:35:16.892794 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:16.892691 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz"] Nov 02 20:35:16.895963 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:16.895931 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-4v4nz"] Nov 02 20:35:17.144934 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:17.144691 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:17.313911 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:17.313869 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9874e15-3cc4-4b17-962a-94bf2d026d94" path="/var/lib/kubelet/pods/a9874e15-3cc4-4b17-962a-94bf2d026d94/volumes" Nov 02 20:35:17.314321 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:17.314308 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c088f7da-23e3-4952-bb72-a9e91456d9a3" path="/var/lib/kubelet/pods/c088f7da-23e3-4952-bb72-a9e91456d9a3/volumes" Nov 02 20:35:17.334143 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:17.334082 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:17.511313 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:17.511269 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:20.515088 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:20.515049 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:25.345184 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:25.345147 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:26.125835 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:26.125781 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:26.550498 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:26.550456 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:28.044856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:28.044821 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:28.399464 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:28.399383 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:30.086648 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:30.086616 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:30.272193 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:30.272119 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:32.024416 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:32.024377 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:32.209722 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:32.209686 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:33.781136 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:35:33.781101 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:35:35.138735 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:35.138695 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:37.355859 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:37.355818 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:38.528155 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:38.528123 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:38.707510 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:38.707473 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:40.234291 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.234252 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:40.544707 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.544610 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj"] Nov 02 20:35:40.545250 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.545230 5561 cpu_manager.go:414] "containerMap: removing container" podUID="c088f7da-23e3-4952-bb72-a9e91456d9a3" containerName="istio-proxy" Nov 02 20:35:40.545380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.545251 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c088f7da-23e3-4952-bb72-a9e91456d9a3" containerName="istio-proxy" Nov 02 20:35:40.545380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.545276 5561 cpu_manager.go:414] "containerMap: removing container" podUID="a9874e15-3cc4-4b17-962a-94bf2d026d94" containerName="istio-proxy" Nov 02 20:35:40.545380 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.545284 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9874e15-3cc4-4b17-962a-94bf2d026d94" containerName="istio-proxy" Nov 02 20:35:40.545538 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.545461 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="c088f7da-23e3-4952-bb72-a9e91456d9a3" containerName="istio-proxy" Nov 02 20:35:40.545538 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.545477 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9874e15-3cc4-4b17-962a-94bf2d026d94" containerName="istio-proxy" Nov 02 20:35:40.548624 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.548602 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.571297 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.571266 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tls-certs\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.571454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.571310 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-dshm\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.571454 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.571390 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-model-cache\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.571546 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.571503 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt6bs\" (UniqueName: \"kubernetes.io/projected/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-kube-api-access-tt6bs\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.571587 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.571550 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tmp-dir\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.571629 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.571613 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-home\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.646261 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.646230 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-65f28f\"/\"kube-root-ca.crt\"" Nov 02 20:35:40.646702 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.646681 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-65f28f\"/\"test-model-65f28f-kserve-self-signed-certs\"" Nov 02 20:35:40.646758 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.646711 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-65f28f\"/\"default-dockercfg-nm2s8\"" Nov 02 20:35:40.646758 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.646717 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-65f28f\"/\"openshift-service-ca.crt\"" Nov 02 20:35:40.648742 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.648693 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj"] Nov 02 20:35:40.673027 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.672997 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt6bs\" (UniqueName: \"kubernetes.io/projected/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-kube-api-access-tt6bs\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.673182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.673049 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tmp-dir\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.673182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.673082 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-home\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.673261 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.673186 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tls-certs\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.673261 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.673220 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-dshm\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.673261 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.673254 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-model-cache\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.673540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.673512 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tmp-dir\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.673580 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.673543 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-home\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.673615 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.673580 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-model-cache\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.675515 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.675492 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-dshm\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.675691 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.675676 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tls-certs\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.683223 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.683192 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:40.726920 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.726865 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt6bs\" (UniqueName: \"kubernetes.io/projected/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-kube-api-access-tt6bs\") pod \"test-model-65f28f-kserve-5c75898bf-rlzhj\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:40.860560 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:40.860465 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:41.245861 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:41.245825 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj"] Nov 02 20:35:41.253202 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:35:41.253175 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6faf63d9_03e2_4f8e_8a04_92d2f6903e74.slice/crio-e541b1dc3b2eaadf4e5f33e7c1305d8993959af4b8e837d271e3f43fc7dcb1d5 WatchSource:0}: Error finding container e541b1dc3b2eaadf4e5f33e7c1305d8993959af4b8e837d271e3f43fc7dcb1d5: Status 404 returned error can't find the container with id e541b1dc3b2eaadf4e5f33e7c1305d8993959af4b8e837d271e3f43fc7dcb1d5 Nov 02 20:35:41.771744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:41.771707 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" event={"ID":"6faf63d9-03e2-4f8e-8a04-92d2f6903e74","Type":"ContainerStarted","Data":"84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8"} Nov 02 20:35:41.771744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:41.771751 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" event={"ID":"6faf63d9-03e2-4f8e-8a04-92d2f6903e74","Type":"ContainerStarted","Data":"e541b1dc3b2eaadf4e5f33e7c1305d8993959af4b8e837d271e3f43fc7dcb1d5"} Nov 02 20:35:41.861167 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:41.861130 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:41.879812 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:41.879764 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" podStartSLOduration=1.879750689 podStartE2EDuration="1.879750689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:41.879239146 +0000 UTC m=+2578.687023407" watchObservedRunningTime="2026-11-02 20:35:41.879750689 +0000 UTC m=+2578.687534951" Nov 02 20:35:44.190816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:44.190779 5561 scope.go:122] "RemoveContainer" containerID="f104acf0f5f2761d809fb129a1a2220a421de38e4bea3a661c06e57e530f50b8" Nov 02 20:35:45.206315 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:45.206204 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:49.741746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:49.741707 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:50.752670 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:50.752632 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:50.861369 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:50.861338 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:50.873919 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:50.873867 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:51.836338 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:51.836303 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:35:53.890245 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:53.890211 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz"] Nov 02 20:35:53.890657 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:53.890401 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" podUID="f6ff587f-ba28-4c80-b51a-11fb3755bad9" containerName="maas-api" containerID="cri-o://dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e" gracePeriod=30 Nov 02 20:35:54.034112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.034075 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:54.154478 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.154456 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:35:54.228343 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.228311 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ff587f-ba28-4c80-b51a-11fb3755bad9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ff587f-ba28-4c80-b51a-11fb3755bad9-openshift-service-ca\") pod \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " Nov 02 20:35:54.228551 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.228410 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ff587f-ba28-4c80-b51a-11fb3755bad9-kube-api-access-n4s47\" (UniqueName: \"kubernetes.io/projected/f6ff587f-ba28-4c80-b51a-11fb3755bad9-kube-api-access-n4s47\") pod \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " Nov 02 20:35:54.228551 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.228509 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-maas-api-tls\") pod \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " Nov 02 20:35:54.228681 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.228553 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-metrics-certs\") pod \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\" (UID: \"f6ff587f-ba28-4c80-b51a-11fb3755bad9\") " Nov 02 20:35:54.228764 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.228723 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ff587f-ba28-4c80-b51a-11fb3755bad9-openshift-service-ca" pod "f6ff587f-ba28-4c80-b51a-11fb3755bad9" (UID: "f6ff587f-ba28-4c80-b51a-11fb3755bad9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:54.229040 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.229016 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ff587f-ba28-4c80-b51a-11fb3755bad9-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:54.230906 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.230842 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ff587f-ba28-4c80-b51a-11fb3755bad9-kube-api-access-n4s47" pod "f6ff587f-ba28-4c80-b51a-11fb3755bad9" (UID: "f6ff587f-ba28-4c80-b51a-11fb3755bad9"). InnerVolumeSpecName "kube-api-access-n4s47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:54.231007 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.230977 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-maas-api-tls" pod "f6ff587f-ba28-4c80-b51a-11fb3755bad9" (UID: "f6ff587f-ba28-4c80-b51a-11fb3755bad9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:35:54.231194 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.231167 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-metrics-certs" pod "f6ff587f-ba28-4c80-b51a-11fb3755bad9" (UID: "f6ff587f-ba28-4c80-b51a-11fb3755bad9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:35:54.330464 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.330426 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:54.330464 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.330462 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ff587f-ba28-4c80-b51a-11fb3755bad9-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:54.330731 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.330480 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4s47\" (UniqueName: \"kubernetes.io/projected/f6ff587f-ba28-4c80-b51a-11fb3755bad9-kube-api-access-n4s47\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:54.846125 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.846088 5561 generic.go:356] "Generic (PLEG): container finished" podID="f6ff587f-ba28-4c80-b51a-11fb3755bad9" containerID="dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e" exitCode=0 Nov 02 20:35:54.846337 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.846147 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" Nov 02 20:35:54.846337 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.846154 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" event={"ID":"f6ff587f-ba28-4c80-b51a-11fb3755bad9","Type":"ContainerDied","Data":"dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e"} Nov 02 20:35:54.846337 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.846200 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz" event={"ID":"f6ff587f-ba28-4c80-b51a-11fb3755bad9","Type":"ContainerDied","Data":"a81c8ba2e0bc45769a76aed0a6742818f4ff54affaff87fd87b6c9a9ca5caec4"} Nov 02 20:35:54.846337 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.846222 5561 scope.go:122] "RemoveContainer" containerID="dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e" Nov 02 20:35:54.858330 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.858301 5561 scope.go:122] "RemoveContainer" containerID="dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e" Nov 02 20:35:54.858912 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:35:54.858853 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e\": container with ID starting with dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e not found: ID does not exist" containerID="dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e" Nov 02 20:35:54.859075 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:54.859047 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e"} err="failed to get container status \"dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e\": rpc error: code = NotFound desc = could not find container \"dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e\": container with ID starting with dd3a294dc3752095d83774f12419463f8c61d83a51b1f448a1e27564654ac74e not found: ID does not exist" Nov 02 20:35:55.004204 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:55.004165 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz"] Nov 02 20:35:55.049987 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:55.049950 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-accdf5-5665cf8b98-8lppz"] Nov 02 20:35:55.312954 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:55.312919 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ff587f-ba28-4c80-b51a-11fb3755bad9" path="/var/lib/kubelet/pods/f6ff587f-ba28-4c80-b51a-11fb3755bad9/volumes" Nov 02 20:35:55.570205 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:55.570118 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:55.755421 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:55.755385 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:55.931060 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:55.931026 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:56.450774 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:35:56.450745 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:00.714145 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:00.714064 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:02.077070 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:02.077036 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:02.312428 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:02.312325 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:02.553150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:02.553116 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:03.779369 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:36:03.779330 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:36:05.796604 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:05.796566 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:05.989757 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:05.989724 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:09.894036 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:09.893996 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:11.347760 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:11.347707 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:11.551786 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:11.551723 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:11.736584 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:11.736548 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:12.248868 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:12.248822 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:16.354304 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:16.354259 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:18.273081 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:18.273046 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:18.447575 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:18.447526 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:18.775528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:18.775473 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:21.242712 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:21.242672 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:23.689959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:23.689924 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c"] Nov 02 20:36:23.841827 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:23.841741 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:25.583679 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:25.583631 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:27.167344 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:27.167310 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:27.448778 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:27.448689 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:27.999102 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:27.999063 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:31.121851 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:31.121815 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:31.297179 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:31.297148 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:32.007474 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:32.007437 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:32.178165 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:32.178134 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:32.352590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:32.352509 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:33.780322 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:36:33.780284 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:36:33.838262 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:33.838208 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:34.018410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:34.018374 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:34.199173 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:34.199130 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:34.627399 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:34.627324 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:37.003771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:37.003719 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:38.710812 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:38.710765 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" containerID="cri-o://ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee" gracePeriod=45 Nov 02 20:36:38.986109 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:38.986079 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:36:39.023658 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.023626 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-tmp-dir\") pod \"3751f81c-d14d-420c-8cc7-273e186114a1\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " Nov 02 20:36:39.023830 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.023708 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-home\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-home\") pod \"3751f81c-d14d-420c-8cc7-273e186114a1\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " Nov 02 20:36:39.023830 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.023738 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-dshm\") pod \"3751f81c-d14d-420c-8cc7-273e186114a1\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " Nov 02 20:36:39.023830 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.023792 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3751f81c-d14d-420c-8cc7-273e186114a1-kube-api-access-wcbd4\" (UniqueName: \"kubernetes.io/projected/3751f81c-d14d-420c-8cc7-273e186114a1-kube-api-access-wcbd4\") pod \"3751f81c-d14d-420c-8cc7-273e186114a1\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " Nov 02 20:36:39.024091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.023851 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-model-cache\") pod \"3751f81c-d14d-420c-8cc7-273e186114a1\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " Nov 02 20:36:39.024091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.023946 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-tmp-dir" pod "3751f81c-d14d-420c-8cc7-273e186114a1" (UID: "3751f81c-d14d-420c-8cc7-273e186114a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:39.024091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.023975 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3751f81c-d14d-420c-8cc7-273e186114a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/3751f81c-d14d-420c-8cc7-273e186114a1-tls-certs\") pod \"3751f81c-d14d-420c-8cc7-273e186114a1\" (UID: \"3751f81c-d14d-420c-8cc7-273e186114a1\") " Nov 02 20:36:39.024091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.024036 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-home" pod "3751f81c-d14d-420c-8cc7-273e186114a1" (UID: "3751f81c-d14d-420c-8cc7-273e186114a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:39.024310 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.024195 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-model-cache" pod "3751f81c-d14d-420c-8cc7-273e186114a1" (UID: "3751f81c-d14d-420c-8cc7-273e186114a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:39.024369 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.024359 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.024425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.024375 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.024425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.024388 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.026098 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.026074 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3751f81c-d14d-420c-8cc7-273e186114a1-tls-certs" pod "3751f81c-d14d-420c-8cc7-273e186114a1" (UID: "3751f81c-d14d-420c-8cc7-273e186114a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:39.026098 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.026080 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-dshm" pod "3751f81c-d14d-420c-8cc7-273e186114a1" (UID: "3751f81c-d14d-420c-8cc7-273e186114a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:39.026270 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.026219 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3751f81c-d14d-420c-8cc7-273e186114a1-kube-api-access-wcbd4" pod "3751f81c-d14d-420c-8cc7-273e186114a1" (UID: "3751f81c-d14d-420c-8cc7-273e186114a1"). InnerVolumeSpecName "kube-api-access-wcbd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:39.088970 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.088935 5561 generic.go:356] "Generic (PLEG): container finished" podID="3751f81c-d14d-420c-8cc7-273e186114a1" containerID="ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee" exitCode=0 Nov 02 20:36:39.089144 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.088985 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" Nov 02 20:36:39.089144 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.088982 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" event={"ID":"3751f81c-d14d-420c-8cc7-273e186114a1","Type":"ContainerDied","Data":"ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee"} Nov 02 20:36:39.089144 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.089098 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" event={"ID":"3751f81c-d14d-420c-8cc7-273e186114a1","Type":"ContainerDied","Data":"416816ae4e46a2e89fb36fdcef5f40b6f37c2c6ae413498bca984d9c09d95800"} Nov 02 20:36:39.089144 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.089114 5561 scope.go:122] "RemoveContainer" containerID="ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee" Nov 02 20:36:39.101782 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.101745 5561 scope.go:122] "RemoveContainer" containerID="ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee" Nov 02 20:36:39.102183 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:36:39.102149 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee\": container with ID starting with ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee not found: ID does not exist" containerID="ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee" Nov 02 20:36:39.102296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.102196 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee"} err="failed to get container status \"ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee\": rpc error: code = NotFound desc = could not find container \"ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee\": container with ID starting with ca3107e52412a2d40591b970b24e4fc6731f142e1b82d63fa3f7b2914d25cbee not found: ID does not exist" Nov 02 20:36:39.125267 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.125217 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcbd4\" (UniqueName: \"kubernetes.io/projected/3751f81c-d14d-420c-8cc7-273e186114a1-kube-api-access-wcbd4\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.125267 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.125260 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3751f81c-d14d-420c-8cc7-273e186114a1-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.125510 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.125277 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3751f81c-d14d-420c-8cc7-273e186114a1-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.250017 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.249927 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c"] Nov 02 20:36:39.254757 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.254725 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c"] Nov 02 20:36:39.313901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.313845 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" path="/var/lib/kubelet/pods/3751f81c-d14d-420c-8cc7-273e186114a1/volumes" Nov 02 20:36:39.545904 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.545809 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz"] Nov 02 20:36:39.551732 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.551700 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p"] Nov 02 20:36:39.566293 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.566255 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5"] Nov 02 20:36:39.749930 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.749893 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:39.938461 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:39.938411 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/distinct-2-w3-2052e3-kserve-7df9595d96-t7q8c" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 02 20:36:41.241565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:41.241526 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:42.134020 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:42.133983 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:45.211368 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.211333 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj"] Nov 02 20:36:45.212018 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.211997 5561 cpu_manager.go:414] "containerMap: removing container" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" Nov 02 20:36:45.212117 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.212019 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" Nov 02 20:36:45.212117 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.212035 5561 cpu_manager.go:414] "containerMap: removing container" podUID="f6ff587f-ba28-4c80-b51a-11fb3755bad9" containerName="maas-api" Nov 02 20:36:45.212117 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.212043 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ff587f-ba28-4c80-b51a-11fb3755bad9" containerName="maas-api" Nov 02 20:36:45.212273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.212162 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="3751f81c-d14d-420c-8cc7-273e186114a1" containerName="main" Nov 02 20:36:45.212273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.212175 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ff587f-ba28-4c80-b51a-11fb3755bad9" containerName="maas-api" Nov 02 20:36:45.216564 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.216543 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.270064 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.270022 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj"] Nov 02 20:36:45.293604 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.293569 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-model-cache\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.293604 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.293608 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-dshm\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.293827 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.293692 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68w9g\" (UniqueName: \"kubernetes.io/projected/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-kube-api-access-68w9g\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.293827 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.293786 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tmp-dir\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.293827 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.293821 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tls-certs\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.293952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.293851 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-home\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.314341 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.314301 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bb78ba\"/\"kube-root-ca.crt\"" Nov 02 20:36:45.314551 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.314382 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bb78ba\"/\"openshift-service-ca.crt\"" Nov 02 20:36:45.314616 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.314587 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bb78ba\"/\"default-dockercfg-5w2qm\"" Nov 02 20:36:45.314666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.314603 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bb78ba\"/\"test-model-bb78ba-kserve-self-signed-certs\"" Nov 02 20:36:45.377330 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.377252 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:45.395535 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.395504 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tmp-dir\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.395699 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.395541 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tls-certs\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.395699 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.395568 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-home\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.395699 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.395636 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-model-cache\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.395699 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.395652 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-dshm\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.395699 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.395693 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68w9g\" (UniqueName: \"kubernetes.io/projected/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-kube-api-access-68w9g\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.396005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.395980 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tmp-dir\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.396069 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.396018 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-home\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.396069 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.396036 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-model-cache\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.397954 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.397927 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-dshm\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.398084 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.398064 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tls-certs\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.449825 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.449778 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68w9g\" (UniqueName: \"kubernetes.io/projected/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-kube-api-access-68w9g\") pod \"test-model-bb78ba-kserve-5588665c75-m7mbj\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.529704 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.529585 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:45.907741 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:45.907706 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj"] Nov 02 20:36:45.915292 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:36:45.915264 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bf23a0b_8cef_49dc_87f8_68ad4f0547b1.slice/crio-d704ef3e9d4d5fd62c1362a2106211506751701a297755d757e6d46c5a49dc4e WatchSource:0}: Error finding container d704ef3e9d4d5fd62c1362a2106211506751701a297755d757e6d46c5a49dc4e: Status 404 returned error can't find the container with id d704ef3e9d4d5fd62c1362a2106211506751701a297755d757e6d46c5a49dc4e Nov 02 20:36:46.129298 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:46.129262 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" event={"ID":"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1","Type":"ContainerStarted","Data":"994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1"} Nov 02 20:36:46.129469 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:46.129309 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" event={"ID":"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1","Type":"ContainerStarted","Data":"d704ef3e9d4d5fd62c1362a2106211506751701a297755d757e6d46c5a49dc4e"} Nov 02 20:36:46.237995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:46.237891 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" podStartSLOduration=1.237862262 podStartE2EDuration="1.237862262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:46.236418842 +0000 UTC m=+2643.044203078" watchObservedRunningTime="2026-11-02 20:36:46.237862262 +0000 UTC m=+2643.045646472" Nov 02 20:36:46.530581 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:46.530505 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:46.764728 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:46.764693 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:50.065919 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:50.055603 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:51.362685 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:51.362642 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:54.573414 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:54.573368 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" containerID="cri-o://48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75" gracePeriod=45 Nov 02 20:36:54.575541 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:54.575511 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" containerID="cri-o://1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44" gracePeriod=45 Nov 02 20:36:54.588343 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:54.588304 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" containerID="cri-o://46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42" gracePeriod=45 Nov 02 20:36:54.897483 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:54.897453 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:36:54.901451 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:54.901427 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:36:54.904975 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:54.904953 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:36:55.021377 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021338 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94af5da2-6687-4956-97d2-ca8ef68b5854-kube-api-access-nrdh7\" (UniqueName: \"kubernetes.io/projected/94af5da2-6687-4956-97d2-ca8ef68b5854-kube-api-access-nrdh7\") pod \"94af5da2-6687-4956-97d2-ca8ef68b5854\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " Nov 02 20:36:55.021561 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021452 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-dshm\") pod \"94af5da2-6687-4956-97d2-ca8ef68b5854\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " Nov 02 20:36:55.021561 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021495 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-dshm\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-dshm\") pod \"787327eb-1c48-4383-b74c-7d468b0cb511\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " Nov 02 20:36:55.021561 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021530 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-tmp-dir\") pod \"787327eb-1c48-4383-b74c-7d468b0cb511\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " Nov 02 20:36:55.021733 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021559 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-home\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-home\") pod \"94af5da2-6687-4956-97d2-ca8ef68b5854\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " Nov 02 20:36:55.021733 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021623 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-dshm\") pod \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " Nov 02 20:36:55.021733 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021654 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-model-cache\") pod \"94af5da2-6687-4956-97d2-ca8ef68b5854\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " Nov 02 20:36:55.021733 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021718 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-model-cache\") pod \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " Nov 02 20:36:55.021994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021752 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-kube-api-access-wj6m2\" (UniqueName: \"kubernetes.io/projected/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-kube-api-access-wj6m2\") pod \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " Nov 02 20:36:55.021994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021783 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94af5da2-6687-4956-97d2-ca8ef68b5854-tls-certs\" (UniqueName: \"kubernetes.io/secret/94af5da2-6687-4956-97d2-ca8ef68b5854-tls-certs\") pod \"94af5da2-6687-4956-97d2-ca8ef68b5854\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " Nov 02 20:36:55.021994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021828 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-tmp-dir\") pod \"94af5da2-6687-4956-97d2-ca8ef68b5854\" (UID: \"94af5da2-6687-4956-97d2-ca8ef68b5854\") " Nov 02 20:36:55.021994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021886 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-model-cache\") pod \"787327eb-1c48-4383-b74c-7d468b0cb511\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " Nov 02 20:36:55.021994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021914 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tls-certs\") pod \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " Nov 02 20:36:55.021994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021950 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-home\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-home\") pod \"787327eb-1c48-4383-b74c-7d468b0cb511\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " Nov 02 20:36:55.022288 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.021999 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/787327eb-1c48-4383-b74c-7d468b0cb511-kube-api-access-mpfpx\" (UniqueName: \"kubernetes.io/projected/787327eb-1c48-4383-b74c-7d468b0cb511-kube-api-access-mpfpx\") pod \"787327eb-1c48-4383-b74c-7d468b0cb511\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " Nov 02 20:36:55.022288 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.022049 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-home\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-home\") pod \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " Nov 02 20:36:55.022288 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.022099 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tmp-dir\") pod \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\" (UID: \"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870\") " Nov 02 20:36:55.022288 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.022141 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/787327eb-1c48-4383-b74c-7d468b0cb511-tls-certs\" (UniqueName: \"kubernetes.io/secret/787327eb-1c48-4383-b74c-7d468b0cb511-tls-certs\") pod \"787327eb-1c48-4383-b74c-7d468b0cb511\" (UID: \"787327eb-1c48-4383-b74c-7d468b0cb511\") " Nov 02 20:36:55.023716 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.023681 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-tmp-dir" pod "787327eb-1c48-4383-b74c-7d468b0cb511" (UID: "787327eb-1c48-4383-b74c-7d468b0cb511"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:55.023847 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.023691 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-home" pod "94af5da2-6687-4956-97d2-ca8ef68b5854" (UID: "94af5da2-6687-4956-97d2-ca8ef68b5854"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:55.024017 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.023992 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-tmp-dir" pod "94af5da2-6687-4956-97d2-ca8ef68b5854" (UID: "94af5da2-6687-4956-97d2-ca8ef68b5854"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:55.024220 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.024196 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-model-cache" pod "787327eb-1c48-4383-b74c-7d468b0cb511" (UID: "787327eb-1c48-4383-b74c-7d468b0cb511"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:55.024510 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.024485 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-home" pod "787327eb-1c48-4383-b74c-7d468b0cb511" (UID: "787327eb-1c48-4383-b74c-7d468b0cb511"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:55.024724 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.024703 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-home" pod "ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" (UID: "ef8d55e1-2ef6-4a45-b4f2-98aefb12b870"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:55.024952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.024928 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-model-cache" pod "94af5da2-6687-4956-97d2-ca8ef68b5854" (UID: "94af5da2-6687-4956-97d2-ca8ef68b5854"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:55.027092 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.027064 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94af5da2-6687-4956-97d2-ca8ef68b5854-tls-certs" pod "94af5da2-6687-4956-97d2-ca8ef68b5854" (UID: "94af5da2-6687-4956-97d2-ca8ef68b5854"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:55.027487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.027465 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tmp-dir" pod "ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" (UID: "ef8d55e1-2ef6-4a45-b4f2-98aefb12b870"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:55.027768 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.027631 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-model-cache" pod "ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" (UID: "ef8d55e1-2ef6-4a45-b4f2-98aefb12b870"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:55.027932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.027894 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-dshm" pod "ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" (UID: "ef8d55e1-2ef6-4a45-b4f2-98aefb12b870"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:55.028828 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.028802 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-kube-api-access-wj6m2" pod "ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" (UID: "ef8d55e1-2ef6-4a45-b4f2-98aefb12b870"). InnerVolumeSpecName "kube-api-access-wj6m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:55.030199 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.030150 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-dshm" pod "94af5da2-6687-4956-97d2-ca8ef68b5854" (UID: "94af5da2-6687-4956-97d2-ca8ef68b5854"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:55.030199 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.030164 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-dshm" pod "787327eb-1c48-4383-b74c-7d468b0cb511" (UID: "787327eb-1c48-4383-b74c-7d468b0cb511"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:55.030199 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.030171 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94af5da2-6687-4956-97d2-ca8ef68b5854-kube-api-access-nrdh7" pod "94af5da2-6687-4956-97d2-ca8ef68b5854" (UID: "94af5da2-6687-4956-97d2-ca8ef68b5854"). InnerVolumeSpecName "kube-api-access-nrdh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:55.030351 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.030214 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tls-certs" pod "ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" (UID: "ef8d55e1-2ef6-4a45-b4f2-98aefb12b870"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:55.030351 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.030225 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/787327eb-1c48-4383-b74c-7d468b0cb511-tls-certs" pod "787327eb-1c48-4383-b74c-7d468b0cb511" (UID: "787327eb-1c48-4383-b74c-7d468b0cb511"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:55.031493 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.031473 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787327eb-1c48-4383-b74c-7d468b0cb511-kube-api-access-mpfpx" pod "787327eb-1c48-4383-b74c-7d468b0cb511" (UID: "787327eb-1c48-4383-b74c-7d468b0cb511"). InnerVolumeSpecName "kube-api-access-mpfpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:55.123937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.123824 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.123937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.123861 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.123937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.123895 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.123937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.123912 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.123937 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.123925 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.123942 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.123958 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj6m2\" (UniqueName: \"kubernetes.io/projected/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-kube-api-access-wj6m2\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.123974 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94af5da2-6687-4956-97d2-ca8ef68b5854-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.123988 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.124001 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.124014 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.124027 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787327eb-1c48-4383-b74c-7d468b0cb511-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.124040 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpfpx\" (UniqueName: \"kubernetes.io/projected/787327eb-1c48-4383-b74c-7d468b0cb511-kube-api-access-mpfpx\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.124054 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.124066 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.124079 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787327eb-1c48-4383-b74c-7d468b0cb511-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.124091 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrdh7\" (UniqueName: \"kubernetes.io/projected/94af5da2-6687-4956-97d2-ca8ef68b5854-kube-api-access-nrdh7\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.124317 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.124104 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94af5da2-6687-4956-97d2-ca8ef68b5854-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.183751 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.183717 5561 generic.go:356] "Generic (PLEG): container finished" podID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerID="46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42" exitCode=0 Nov 02 20:36:55.183983 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.183763 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" Nov 02 20:36:55.183983 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.183797 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" event={"ID":"94af5da2-6687-4956-97d2-ca8ef68b5854","Type":"ContainerDied","Data":"46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42"} Nov 02 20:36:55.183983 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.183847 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" event={"ID":"94af5da2-6687-4956-97d2-ca8ef68b5854","Type":"ContainerDied","Data":"48ce66cbb30892a802f89730ebfc5c111836cf12b733cbd29fedbf3237ecc6fa"} Nov 02 20:36:55.183983 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.183870 5561 scope.go:122] "RemoveContainer" containerID="46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42" Nov 02 20:36:55.185219 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.185200 5561 generic.go:356] "Generic (PLEG): container finished" podID="787327eb-1c48-4383-b74c-7d468b0cb511" containerID="1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44" exitCode=0 Nov 02 20:36:55.185313 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.185226 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" event={"ID":"787327eb-1c48-4383-b74c-7d468b0cb511","Type":"ContainerDied","Data":"1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44"} Nov 02 20:36:55.185313 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.185257 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" Nov 02 20:36:55.185313 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.185263 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p" event={"ID":"787327eb-1c48-4383-b74c-7d468b0cb511","Type":"ContainerDied","Data":"e8a81d727743de27940d2712d7ed3140dc8042233c72b9fdfe8a80100cdaf158"} Nov 02 20:36:55.186686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.186664 5561 generic.go:356] "Generic (PLEG): container finished" podID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerID="48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75" exitCode=0 Nov 02 20:36:55.186780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.186747 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" event={"ID":"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870","Type":"ContainerDied","Data":"48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75"} Nov 02 20:36:55.186780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.186774 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" event={"ID":"ef8d55e1-2ef6-4a45-b4f2-98aefb12b870","Type":"ContainerDied","Data":"66dd6647f58caad21198485a01c4dde716f7a1188c098b465917ca0109141a18"} Nov 02 20:36:55.186958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.186829 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz" Nov 02 20:36:55.196039 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.196016 5561 scope.go:122] "RemoveContainer" containerID="46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42" Nov 02 20:36:55.196328 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:36:55.196311 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42\": container with ID starting with 46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42 not found: ID does not exist" containerID="46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42" Nov 02 20:36:55.196391 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.196338 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42"} err="failed to get container status \"46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42\": rpc error: code = NotFound desc = could not find container \"46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42\": container with ID starting with 46dee1d20d34cd1163109342447e48c68691b385abb60eeb418901bbdec9ab42 not found: ID does not exist" Nov 02 20:36:55.196391 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.196354 5561 scope.go:122] "RemoveContainer" containerID="1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44" Nov 02 20:36:55.205342 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.205276 5561 scope.go:122] "RemoveContainer" containerID="1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44" Nov 02 20:36:55.205619 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:36:55.205595 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44\": container with ID starting with 1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44 not found: ID does not exist" containerID="1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44" Nov 02 20:36:55.205735 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.205626 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44"} err="failed to get container status \"1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44\": rpc error: code = NotFound desc = could not find container \"1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44\": container with ID starting with 1f0b3bed8b203994005b574c9f2dbaecc824104973032c40a206516200289b44 not found: ID does not exist" Nov 02 20:36:55.205735 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.205649 5561 scope.go:122] "RemoveContainer" containerID="48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75" Nov 02 20:36:55.214774 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.214751 5561 scope.go:122] "RemoveContainer" containerID="48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75" Nov 02 20:36:55.215131 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:36:55.215102 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75\": container with ID starting with 48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75 not found: ID does not exist" containerID="48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75" Nov 02 20:36:55.215221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.215143 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75"} err="failed to get container status \"48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75\": rpc error: code = NotFound desc = could not find container \"48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75\": container with ID starting with 48d91984017fe5673f050ce2a5018184c42cb1c77db120cbedbd9ed1ea3ccf75 not found: ID does not exist" Nov 02 20:36:55.267830 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:36:55.267786 5561 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod787327eb_1c48_4383_b74c_7d468b0cb511.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod787327eb_1c48_4383_b74c_7d468b0cb511.slice/crio-e8a81d727743de27940d2712d7ed3140dc8042233c72b9fdfe8a80100cdaf158\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef8d55e1_2ef6_4a45_b4f2_98aefb12b870.slice/crio-66dd6647f58caad21198485a01c4dde716f7a1188c098b465917ca0109141a18\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94af5da2_6687_4956_97d2_ca8ef68b5854.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94af5da2_6687_4956_97d2_ca8ef68b5854.slice/crio-48ce66cbb30892a802f89730ebfc5c111836cf12b733cbd29fedbf3237ecc6fa\": RecentStats: unable to find data in memory cache]" Nov 02 20:36:55.331651 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.331312 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p"] Nov 02 20:36:55.339583 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.339555 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/distinct-w6-982b8a-kserve-6cf454f54f-hph4p"] Nov 02 20:36:55.444863 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.444828 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz"] Nov 02 20:36:55.448682 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.448652 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/distinct-2-w6-982b8a-kserve-5cd56486-qmbsz"] Nov 02 20:36:55.530176 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.530142 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:55.543038 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.543005 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:55.552980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.552951 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5"] Nov 02 20:36:55.557190 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.557159 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5"] Nov 02 20:36:55.600789 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.600759 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p"] Nov 02 20:36:55.601285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.601268 5561 cpu_manager.go:414] "containerMap: removing container" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" Nov 02 20:36:55.601345 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.601286 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" Nov 02 20:36:55.601345 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.601296 5561 cpu_manager.go:414] "containerMap: removing container" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" Nov 02 20:36:55.601345 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.601301 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" Nov 02 20:36:55.601345 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.601315 5561 cpu_manager.go:414] "containerMap: removing container" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" Nov 02 20:36:55.601345 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.601321 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" Nov 02 20:36:55.601506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.601422 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" Nov 02 20:36:55.601506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.601428 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" containerName="main" Nov 02 20:36:55.601506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.601435 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" containerName="main" Nov 02 20:36:55.606118 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.606095 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.658263 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.658224 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p"] Nov 02 20:36:55.658447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.658378 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2b3b9e\"/\"openshift-service-ca.crt\"" Nov 02 20:36:55.658447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.658393 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2b3b9e\"/\"kube-root-ca.crt\"" Nov 02 20:36:55.658447 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.658422 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2b3b9e\"/\"test-model-2b3b9e-kserve-self-signed-certs\"" Nov 02 20:36:55.658578 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.658467 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2b3b9e\"/\"default-dockercfg-fvs8l\"" Nov 02 20:36:55.730946 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.730849 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tls-certs\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.731085 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.730971 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-model-cache\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.731085 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.731015 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tmp-dir\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.731176 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.731159 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-dshm\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.731226 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.731213 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-home\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.731292 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.731274 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6dmr\" (UniqueName: \"kubernetes.io/projected/de249c08-26bd-4582-b4c6-dce05a1ae3b5-kube-api-access-x6dmr\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.742139 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.742077 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:55.832376 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.832342 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-dshm\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.832573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.832395 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-home\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.832573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.832417 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6dmr\" (UniqueName: \"kubernetes.io/projected/de249c08-26bd-4582-b4c6-dce05a1ae3b5-kube-api-access-x6dmr\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.832573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.832468 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tls-certs\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.832573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.832501 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-model-cache\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.832573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.832537 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tmp-dir\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.832950 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.832922 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-home\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.833030 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.832963 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tmp-dir\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.833084 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.833045 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-model-cache\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.835145 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.835116 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-dshm\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.835402 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.835378 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tls-certs\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.876677 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.876619 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-982b8a/simulator-w6-982b8a-kserve-7b5d974c5-sw8c5" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:55.888590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.888559 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6dmr\" (UniqueName: \"kubernetes.io/projected/de249c08-26bd-4582-b4c6-dce05a1ae3b5-kube-api-access-x6dmr\") pod \"test-model-2b3b9e-kserve-6f654f955c-25r8p\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.918175 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.918130 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:36:55.998995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:55.998122 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:56.184571 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:56.184532 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:56.209502 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:56.209463 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:36:56.302334 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:56.302251 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p"] Nov 02 20:36:56.309686 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:36:56.309653 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde249c08_26bd_4582_b4c6_dce05a1ae3b5.slice/crio-062512e06016491edaeee3521b6274f21d5ede233d96a9c15e40c801e287955d WatchSource:0}: Error finding container 062512e06016491edaeee3521b6274f21d5ede233d96a9c15e40c801e287955d: Status 404 returned error can't find the container with id 062512e06016491edaeee3521b6274f21d5ede233d96a9c15e40c801e287955d Nov 02 20:36:56.532165 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:56.532125 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:57.204427 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:57.204389 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" event={"ID":"de249c08-26bd-4582-b4c6-dce05a1ae3b5","Type":"ContainerStarted","Data":"5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe"} Nov 02 20:36:57.204427 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:57.204433 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" event={"ID":"de249c08-26bd-4582-b4c6-dce05a1ae3b5","Type":"ContainerStarted","Data":"062512e06016491edaeee3521b6274f21d5ede233d96a9c15e40c801e287955d"} Nov 02 20:36:57.312692 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:57.312639 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" podStartSLOduration=2.312625179 podStartE2EDuration="2.312625179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:57.311700423 +0000 UTC m=+2654.119484638" watchObservedRunningTime="2026-11-02 20:36:57.312625179 +0000 UTC m=+2654.120409395" Nov 02 20:36:57.313323 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:57.313302 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="787327eb-1c48-4383-b74c-7d468b0cb511" path="/var/lib/kubelet/pods/787327eb-1c48-4383-b74c-7d468b0cb511/volumes" Nov 02 20:36:57.313731 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:57.313718 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94af5da2-6687-4956-97d2-ca8ef68b5854" path="/var/lib/kubelet/pods/94af5da2-6687-4956-97d2-ca8ef68b5854/volumes" Nov 02 20:36:57.314122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:57.314108 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8d55e1-2ef6-4a45-b4f2-98aefb12b870" path="/var/lib/kubelet/pods/ef8d55e1-2ef6-4a45-b4f2-98aefb12b870/volumes" Nov 02 20:36:57.918760 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:36:57.918728 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:37:00.072531 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:00.072488 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:00.256722 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:00.256677 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:00.826030 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:00.819207 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:01.445090 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:01.445050 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:03.781016 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:37:03.780981 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:37:05.426039 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:05.426008 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:05.918554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:05.918531 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:37:05.930931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:05.930897 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:37:06.273583 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:06.273495 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:37:07.252322 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:07.252286 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:09.967050 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:09.967019 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:12.813832 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:12.813791 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:14.656855 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:14.656820 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:15.613039 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:15.613002 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:15.782288 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:15.782256 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:16.302470 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:16.302424 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:18.379829 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:18.379790 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:18.600812 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:18.600771 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:19.174443 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:19.174399 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:22.138383 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:22.138341 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:23.765288 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:23.765232 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:28.708460 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:28.708377 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:29.427130 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:29.427099 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:32.841855 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:32.841755 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:33.778513 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:37:33.778467 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:37:34.299517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:34.299482 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:34.581579 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:34.581496 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:34.752396 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:34.752364 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:35.188792 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:35.188753 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:39.243686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:39.243641 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:40.756746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:40.756702 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:40.932718 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:40.932661 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:41.042131 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:41.042054 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9"] Nov 02 20:37:41.042305 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:41.042269 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" containerID="cri-o://119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44" gracePeriod=30 Nov 02 20:37:41.118226 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:41.118194 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:43.404578 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:43.404543 5561 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:37:43.405070 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:43.404608 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:37:43.820361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:43.820281 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:46.311102 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.311037 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:37:46.401826 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.401792 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-kube-api-access-pg974\" (UniqueName: \"kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-kube-api-access-pg974\") pod \"386e14bc-708a-4130-9b7e-976e11cc5e24\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " Nov 02 20:37:46.402027 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.401956 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-envoy\") pod \"386e14bc-708a-4130-9b7e-976e11cc5e24\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " Nov 02 20:37:46.402027 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.401988 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386e14bc-708a-4130-9b7e-976e11cc5e24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386e14bc-708a-4130-9b7e-976e11cc5e24-istiod-ca-cert\") pod \"386e14bc-708a-4130-9b7e-976e11cc5e24\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " Nov 02 20:37:46.402153 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402028 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-socket\") pod \"386e14bc-708a-4130-9b7e-976e11cc5e24\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " Nov 02 20:37:46.402153 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402064 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-data\") pod \"386e14bc-708a-4130-9b7e-976e11cc5e24\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " Nov 02 20:37:46.402153 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402100 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-credential-socket\") pod \"386e14bc-708a-4130-9b7e-976e11cc5e24\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " Nov 02 20:37:46.402153 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402133 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-podinfo\") pod \"386e14bc-708a-4130-9b7e-976e11cc5e24\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " Nov 02 20:37:46.402351 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402248 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-token\" (UniqueName: \"kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-token\") pod \"386e14bc-708a-4130-9b7e-976e11cc5e24\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " Nov 02 20:37:46.402351 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402320 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-certs\") pod \"386e14bc-708a-4130-9b7e-976e11cc5e24\" (UID: \"386e14bc-708a-4130-9b7e-976e11cc5e24\") " Nov 02 20:37:46.402480 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402381 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-socket" pod "386e14bc-708a-4130-9b7e-976e11cc5e24" (UID: "386e14bc-708a-4130-9b7e-976e11cc5e24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:46.402480 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402397 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-data" pod "386e14bc-708a-4130-9b7e-976e11cc5e24" (UID: "386e14bc-708a-4130-9b7e-976e11cc5e24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:46.402480 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402462 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386e14bc-708a-4130-9b7e-976e11cc5e24-istiod-ca-cert" pod "386e14bc-708a-4130-9b7e-976e11cc5e24" (UID: "386e14bc-708a-4130-9b7e-976e11cc5e24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:46.402700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402676 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-credential-socket" pod "386e14bc-708a-4130-9b7e-976e11cc5e24" (UID: "386e14bc-708a-4130-9b7e-976e11cc5e24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:46.402700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402687 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-certs" pod "386e14bc-708a-4130-9b7e-976e11cc5e24" (UID: "386e14bc-708a-4130-9b7e-976e11cc5e24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:46.402852 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402833 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.402943 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402856 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386e14bc-708a-4130-9b7e-976e11cc5e24-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.402943 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402891 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.402943 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402907 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.402943 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.402920 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.404272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.404246 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-envoy" pod "386e14bc-708a-4130-9b7e-976e11cc5e24" (UID: "386e14bc-708a-4130-9b7e-976e11cc5e24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:46.404468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.404447 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-token" pod "386e14bc-708a-4130-9b7e-976e11cc5e24" (UID: "386e14bc-708a-4130-9b7e-976e11cc5e24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:46.404562 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.404545 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-kube-api-access-pg974" pod "386e14bc-708a-4130-9b7e-976e11cc5e24" (UID: "386e14bc-708a-4130-9b7e-976e11cc5e24"). InnerVolumeSpecName "kube-api-access-pg974". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:46.404643 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.404629 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-podinfo" pod "386e14bc-708a-4130-9b7e-976e11cc5e24" (UID: "386e14bc-708a-4130-9b7e-976e11cc5e24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:37:46.493207 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.493166 5561 generic.go:356] "Generic (PLEG): container finished" podID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerID="119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44" exitCode=0 Nov 02 20:37:46.493372 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.493232 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" event={"ID":"386e14bc-708a-4130-9b7e-976e11cc5e24","Type":"ContainerDied","Data":"119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44"} Nov 02 20:37:46.493372 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.493273 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" Nov 02 20:37:46.493372 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.493320 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9" event={"ID":"386e14bc-708a-4130-9b7e-976e11cc5e24","Type":"ContainerDied","Data":"4a6ee2c4afb2ea2c6cc0f79bb011f051b252deff2fe21e1ac1daeb8ad2300acb"} Nov 02 20:37:46.493372 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.493351 5561 scope.go:122] "RemoveContainer" containerID="119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44" Nov 02 20:37:46.503729 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.503703 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.503833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.503735 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.503833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.503751 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg974\" (UniqueName: \"kubernetes.io/projected/386e14bc-708a-4130-9b7e-976e11cc5e24-kube-api-access-pg974\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.503833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.503767 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386e14bc-708a-4130-9b7e-976e11cc5e24-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.504647 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.504627 5561 scope.go:122] "RemoveContainer" containerID="9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0" Nov 02 20:37:46.513926 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.513906 5561 scope.go:122] "RemoveContainer" containerID="119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44" Nov 02 20:37:46.514221 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:37:46.514201 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44\": container with ID starting with 119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44 not found: ID does not exist" containerID="119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44" Nov 02 20:37:46.514272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.514231 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44"} err="failed to get container status \"119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44\": rpc error: code = NotFound desc = could not find container \"119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44\": container with ID starting with 119f81e238270cd95873fa1cb9246058d16ee44b4c4414e43154b1ed4e899f44 not found: ID does not exist" Nov 02 20:37:46.514272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.514250 5561 scope.go:122] "RemoveContainer" containerID="9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0" Nov 02 20:37:46.514514 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:37:46.514489 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0\": container with ID starting with 9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0 not found: ID does not exist" containerID="9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0" Nov 02 20:37:46.514613 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.514517 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0"} err="failed to get container status \"9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0\": rpc error: code = NotFound desc = could not find container \"9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0\": container with ID starting with 9297d06304f57f3626ba4c696134f8aef9870f466cefb85e0c0f2c6a0ecaf6e0 not found: ID does not exist" Nov 02 20:37:46.650364 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.650277 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9"] Nov 02 20:37:46.652772 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:46.652741 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-gcbl9"] Nov 02 20:37:47.313457 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.313423 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" path="/var/lib/kubelet/pods/386e14bc-708a-4130-9b7e-976e11cc5e24/volumes" Nov 02 20:37:47.613276 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.613183 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr"] Nov 02 20:37:47.613460 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.613435 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" podUID="429ad3de-f074-44d6-83ac-b8cf9ce545cd" containerName="maas-api" containerID="cri-o://800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe" gracePeriod=30 Nov 02 20:37:47.752182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.752144 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:47.876375 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.876318 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:37:47.920481 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.920438 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-metrics-certs\") pod \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " Nov 02 20:37:47.920640 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.920531 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-maas-api-tls\") pod \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " Nov 02 20:37:47.920640 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.920615 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429ad3de-f074-44d6-83ac-b8cf9ce545cd-kube-api-access-gbvfd\" (UniqueName: \"kubernetes.io/projected/429ad3de-f074-44d6-83ac-b8cf9ce545cd-kube-api-access-gbvfd\") pod \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " Nov 02 20:37:47.920800 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.920782 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429ad3de-f074-44d6-83ac-b8cf9ce545cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/429ad3de-f074-44d6-83ac-b8cf9ce545cd-openshift-service-ca\") pod \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\" (UID: \"429ad3de-f074-44d6-83ac-b8cf9ce545cd\") " Nov 02 20:37:47.921190 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.921163 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429ad3de-f074-44d6-83ac-b8cf9ce545cd-openshift-service-ca" pod "429ad3de-f074-44d6-83ac-b8cf9ce545cd" (UID: "429ad3de-f074-44d6-83ac-b8cf9ce545cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:47.921309 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.921288 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/429ad3de-f074-44d6-83ac-b8cf9ce545cd-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:47.922627 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.922606 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-metrics-certs" pod "429ad3de-f074-44d6-83ac-b8cf9ce545cd" (UID: "429ad3de-f074-44d6-83ac-b8cf9ce545cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:47.923027 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.923008 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-maas-api-tls" pod "429ad3de-f074-44d6-83ac-b8cf9ce545cd" (UID: "429ad3de-f074-44d6-83ac-b8cf9ce545cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:47.923165 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:47.923148 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429ad3de-f074-44d6-83ac-b8cf9ce545cd-kube-api-access-gbvfd" pod "429ad3de-f074-44d6-83ac-b8cf9ce545cd" (UID: "429ad3de-f074-44d6-83ac-b8cf9ce545cd"). InnerVolumeSpecName "kube-api-access-gbvfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:48.022402 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:48.022369 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:48.022402 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:48.022395 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbvfd\" (UniqueName: \"kubernetes.io/projected/429ad3de-f074-44d6-83ac-b8cf9ce545cd-kube-api-access-gbvfd\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:48.022402 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:48.022405 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/429ad3de-f074-44d6-83ac-b8cf9ce545cd-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:48.509735 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:48.509694 5561 generic.go:356] "Generic (PLEG): container finished" podID="429ad3de-f074-44d6-83ac-b8cf9ce545cd" containerID="800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe" exitCode=0 Nov 02 20:37:48.510263 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:48.509739 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" event={"ID":"429ad3de-f074-44d6-83ac-b8cf9ce545cd","Type":"ContainerDied","Data":"800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe"} Nov 02 20:37:48.510263 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:48.509799 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" event={"ID":"429ad3de-f074-44d6-83ac-b8cf9ce545cd","Type":"ContainerDied","Data":"406cd3bd9fa09426611002d0e89a2a352ff00263799ef2b9f2fa037ec935b014"} Nov 02 20:37:48.510263 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:48.509825 5561 scope.go:122] "RemoveContainer" containerID="800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe" Nov 02 20:37:48.510510 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:48.510481 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr" Nov 02 20:37:48.524387 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:48.524355 5561 scope.go:122] "RemoveContainer" containerID="800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe" Nov 02 20:37:48.524929 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:37:48.524866 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe\": container with ID starting with 800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe not found: ID does not exist" containerID="800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe" Nov 02 20:37:48.525038 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:48.524939 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe"} err="failed to get container status \"800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe\": rpc error: code = NotFound desc = could not find container \"800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe\": container with ID starting with 800ae0a59e1158f3b3ab24d1a2d6653c28397ae507a1e40b5927290318ba91fe not found: ID does not exist" Nov 02 20:37:48.666624 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:48.666590 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr"] Nov 02 20:37:48.714244 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:48.714209 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-982b8a-f7d88967c-prvhr"] Nov 02 20:37:49.252829 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:49.252782 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:49.313771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:49.313736 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429ad3de-f074-44d6-83ac-b8cf9ce545cd" path="/var/lib/kubelet/pods/429ad3de-f074-44d6-83ac-b8cf9ce545cd/volumes" Nov 02 20:37:49.451761 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:49.451700 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:49.627627 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:49.627536 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:50.135441 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:50.135403 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:54.148076 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:54.147976 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:55.882190 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:55.882150 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:56.062414 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:56.062377 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:56.162353 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:56.162323 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt"] Nov 02 20:37:56.162661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:56.162552 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" podUID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" containerName="istio-proxy" containerID="cri-o://5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9" gracePeriod=30 Nov 02 20:37:56.249912 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:56.249854 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:56.428848 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:56.428770 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:58.651257 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:37:58.651222 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:01.426735 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.426675 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:38:01.568702 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.568671 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-certs\") pod \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " Nov 02 20:38:01.568913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.568715 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-podinfo\") pod \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " Nov 02 20:38:01.568913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.568755 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-socket\") pod \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " Nov 02 20:38:01.568913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.568780 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istiod-ca-cert\") pod \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " Nov 02 20:38:01.568913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.568805 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-credential-socket\") pod \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " Nov 02 20:38:01.568913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.568832 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-data\") pod \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " Nov 02 20:38:01.569193 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.569117 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-certs" pod "e3e2f75e-18ea-4b40-bada-758f81aa07b0" (UID: "e3e2f75e-18ea-4b40-bada-758f81aa07b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:01.569251 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.569219 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-credential-socket" pod "e3e2f75e-18ea-4b40-bada-758f81aa07b0" (UID: "e3e2f75e-18ea-4b40-bada-758f81aa07b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:01.569354 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.569332 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istiod-ca-cert" pod "e3e2f75e-18ea-4b40-bada-758f81aa07b0" (UID: "e3e2f75e-18ea-4b40-bada-758f81aa07b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:01.569420 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.569358 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-data" pod "e3e2f75e-18ea-4b40-bada-758f81aa07b0" (UID: "e3e2f75e-18ea-4b40-bada-758f81aa07b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:01.569547 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.569517 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-kube-api-access-xgw6r\" (UniqueName: \"kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-kube-api-access-xgw6r\") pod \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " Nov 02 20:38:01.569547 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.569530 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-socket" pod "e3e2f75e-18ea-4b40-bada-758f81aa07b0" (UID: "e3e2f75e-18ea-4b40-bada-758f81aa07b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:01.569734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.569584 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-envoy\") pod \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " Nov 02 20:38:01.569734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.569701 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-token\" (UniqueName: \"kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-token\") pod \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\" (UID: \"e3e2f75e-18ea-4b40-bada-758f81aa07b0\") " Nov 02 20:38:01.570178 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.570154 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:01.570281 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.570182 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:01.570281 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.570191 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:01.570281 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.570200 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:01.570281 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.570208 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:01.571357 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.571320 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-kube-api-access-xgw6r" pod "e3e2f75e-18ea-4b40-bada-758f81aa07b0" (UID: "e3e2f75e-18ea-4b40-bada-758f81aa07b0"). InnerVolumeSpecName "kube-api-access-xgw6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:01.571663 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.571649 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-podinfo" pod "e3e2f75e-18ea-4b40-bada-758f81aa07b0" (UID: "e3e2f75e-18ea-4b40-bada-758f81aa07b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:38:01.571778 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.571764 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-token" pod "e3e2f75e-18ea-4b40-bada-758f81aa07b0" (UID: "e3e2f75e-18ea-4b40-bada-758f81aa07b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:01.572275 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.572260 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-envoy" pod "e3e2f75e-18ea-4b40-bada-758f81aa07b0" (UID: "e3e2f75e-18ea-4b40-bada-758f81aa07b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:01.584224 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.584192 5561 generic.go:356] "Generic (PLEG): container finished" podID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" containerID="5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9" exitCode=0 Nov 02 20:38:01.584346 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.584236 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" Nov 02 20:38:01.584346 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.584279 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" event={"ID":"e3e2f75e-18ea-4b40-bada-758f81aa07b0","Type":"ContainerDied","Data":"5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9"} Nov 02 20:38:01.584459 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.584346 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt" event={"ID":"e3e2f75e-18ea-4b40-bada-758f81aa07b0","Type":"ContainerDied","Data":"5d5494c9c0bd0eaf8cc329a71cf25579619e900edb32606bf4f8262273e0731f"} Nov 02 20:38:01.584459 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.584380 5561 scope.go:122] "RemoveContainer" containerID="5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9" Nov 02 20:38:01.595176 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.595139 5561 scope.go:122] "RemoveContainer" containerID="5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9" Nov 02 20:38:01.595441 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:38:01.595422 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9\": container with ID starting with 5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9 not found: ID does not exist" containerID="5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9" Nov 02 20:38:01.595490 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.595450 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9"} err="failed to get container status \"5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9\": rpc error: code = NotFound desc = could not find container \"5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9\": container with ID starting with 5ad1ee091d3a7db9f5b6c2239c13667ddfc02da1e99b2b5d8f357e69b62892f9 not found: ID does not exist" Nov 02 20:38:01.671672 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.671627 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:01.671672 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.671661 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:01.671672 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.671671 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgw6r\" (UniqueName: \"kubernetes.io/projected/e3e2f75e-18ea-4b40-bada-758f81aa07b0-kube-api-access-xgw6r\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:01.671946 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.671681 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e2f75e-18ea-4b40-bada-758f81aa07b0-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:01.740565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.740483 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt"] Nov 02 20:38:01.743909 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:01.743870 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-wnkwt"] Nov 02 20:38:03.126116 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:03.126069 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:03.313136 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:03.313095 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" path="/var/lib/kubelet/pods/e3e2f75e-18ea-4b40-bada-758f81aa07b0/volumes" Nov 02 20:38:03.779765 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:38:03.779726 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:38:07.713972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:07.713930 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:09.188736 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.188702 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s"] Nov 02 20:38:09.189144 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.189120 5561 cpu_manager.go:414] "containerMap: removing container" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" Nov 02 20:38:09.189144 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.189131 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" Nov 02 20:38:09.189144 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.189144 5561 cpu_manager.go:414] "containerMap: removing container" podUID="429ad3de-f074-44d6-83ac-b8cf9ce545cd" containerName="maas-api" Nov 02 20:38:09.189248 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.189149 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429ad3de-f074-44d6-83ac-b8cf9ce545cd" containerName="maas-api" Nov 02 20:38:09.189248 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.189190 5561 cpu_manager.go:414] "containerMap: removing container" podUID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" containerName="istio-proxy" Nov 02 20:38:09.189248 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.189195 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" containerName="istio-proxy" Nov 02 20:38:09.189343 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.189256 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="429ad3de-f074-44d6-83ac-b8cf9ce545cd" containerName="maas-api" Nov 02 20:38:09.189343 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.189269 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" Nov 02 20:38:09.189343 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.189277 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3e2f75e-18ea-4b40-bada-758f81aa07b0" containerName="istio-proxy" Nov 02 20:38:09.189343 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.189284 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" Nov 02 20:38:09.194049 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.194027 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.246254 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.246220 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s"] Nov 02 20:38:09.252669 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.252639 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-home\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.252829 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.252757 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tls-certs\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.252829 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.252810 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tmp-dir\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.252945 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.252830 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-model-cache\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.252945 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.252856 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7lz4\" (UniqueName: \"kubernetes.io/projected/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-kube-api-access-k7lz4\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.253084 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.253056 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-dshm\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.291673 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.291640 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0d9ec0\"/\"openshift-service-ca.crt\"" Nov 02 20:38:09.291869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.291678 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0d9ec0\"/\"kube-root-ca.crt\"" Nov 02 20:38:09.291869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.291726 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0d9ec0\"/\"default-dockercfg-kbxqs\"" Nov 02 20:38:09.291869 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.291764 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0d9ec0\"/\"test-model-0d9ec0-kserve-self-signed-certs\"" Nov 02 20:38:09.320341 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.320310 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:09.354166 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.354136 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tls-certs\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.354332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.354201 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tmp-dir\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.354332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.354313 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-model-cache\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.354439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.354354 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lz4\" (UniqueName: \"kubernetes.io/projected/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-kube-api-access-k7lz4\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.354439 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.354420 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-dshm\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.354520 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.354482 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-home\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.354847 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.354602 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tmp-dir\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.354847 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.354624 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-model-cache\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.355024 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.354841 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-home\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.357143 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.357116 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-dshm\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.357371 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.357355 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tls-certs\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.408856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.408818 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7lz4\" (UniqueName: \"kubernetes.io/projected/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-kube-api-access-k7lz4\") pod \"test-model-0d9ec0-kserve-5c75f5bf-kdq8s\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.505956 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.505841 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:09.888731 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:09.888650 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s"] Nov 02 20:38:09.895894 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:38:09.895847 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20cd41d9_5f2d_4ece_a7f7_e62487149e0a.slice/crio-dd23a50145e69ce8f0031978359bb88ecb7f27ea9fc4fd9bdea8a2531cdcd7d6 WatchSource:0}: Error finding container dd23a50145e69ce8f0031978359bb88ecb7f27ea9fc4fd9bdea8a2531cdcd7d6: Status 404 returned error can't find the container with id dd23a50145e69ce8f0031978359bb88ecb7f27ea9fc4fd9bdea8a2531cdcd7d6 Nov 02 20:38:10.643746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:10.643704 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" event={"ID":"20cd41d9-5f2d-4ece-a7f7-e62487149e0a","Type":"ContainerStarted","Data":"0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728"} Nov 02 20:38:10.644220 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:10.643761 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" event={"ID":"20cd41d9-5f2d-4ece-a7f7-e62487149e0a","Type":"ContainerStarted","Data":"dd23a50145e69ce8f0031978359bb88ecb7f27ea9fc4fd9bdea8a2531cdcd7d6"} Nov 02 20:38:10.771464 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:10.771398 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" podStartSLOduration=1.771376913 podStartE2EDuration="1.771376913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:10.770133512 +0000 UTC m=+2727.577917729" watchObservedRunningTime="2026-11-02 20:38:10.771376913 +0000 UTC m=+2727.579161130" Nov 02 20:38:10.888388 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:10.888357 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:11.104553 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:11.104518 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:11.506618 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:11.506576 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:12.278360 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:12.278328 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:16.761907 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:16.761847 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:19.505998 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:19.505962 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:19.506773 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:19.506744 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:19.518706 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:19.518684 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:19.703486 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:19.703456 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:38:21.258605 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:21.258483 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:23.784918 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.784883 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78"] Nov 02 20:38:23.785412 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.785397 5561 cpu_manager.go:414] "containerMap: removing container" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" Nov 02 20:38:23.785468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.785413 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386e14bc-708a-4130-9b7e-976e11cc5e24" containerName="istio-proxy" Nov 02 20:38:23.788674 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.788656 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:23.886162 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.886117 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-aaa3e7\"/\"kube-root-ca.crt\"" Nov 02 20:38:23.886335 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.886304 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-aaa3e7\"/\"openshift-service-ca.crt\"" Nov 02 20:38:23.886407 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.886324 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-aaa3e7\"/\"default-dockercfg-8bhv5\"" Nov 02 20:38:23.886407 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.886353 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-aaa3e7\"/\"test-model-aaa3e7-kserve-self-signed-certs\"" Nov 02 20:38:23.888131 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.888107 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78"] Nov 02 20:38:23.912626 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.912603 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-home\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:23.912718 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.912657 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tls-certs\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:23.912803 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.912781 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9nv2\" (UniqueName: \"kubernetes.io/projected/65cfe3fa-bddf-4613-8987-9ebc09c064b4-kube-api-access-v9nv2\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:23.912920 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.912904 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-model-cache\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:23.912982 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.912927 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tmp-dir\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:23.913028 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.913002 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-dshm\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:23.931855 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:23.931820 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:24.014392 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.014347 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9nv2\" (UniqueName: \"kubernetes.io/projected/65cfe3fa-bddf-4613-8987-9ebc09c064b4-kube-api-access-v9nv2\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.014573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.014435 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-model-cache\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.014573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.014462 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tmp-dir\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.014573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.014516 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-dshm\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.014573 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.014570 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-home\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.014798 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.014638 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tls-certs\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.014962 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.014938 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-model-cache\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.015056 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.014958 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tmp-dir\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.015221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.015204 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-home\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.017450 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.017426 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-dshm\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.017700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.017674 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tls-certs\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.069282 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.069209 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9nv2\" (UniqueName: \"kubernetes.io/projected/65cfe3fa-bddf-4613-8987-9ebc09c064b4-kube-api-access-v9nv2\") pod \"test-model-aaa3e7-kserve-69b4c96d67-ffk78\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.101107 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.101065 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:24.489687 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.489651 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78"] Nov 02 20:38:24.496703 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:38:24.496676 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65cfe3fa_bddf_4613_8987_9ebc09c064b4.slice/crio-c6090eedba752c4c65fb5912e1fdb9ad3d891ceeadb18be21092b75dabf70213 WatchSource:0}: Error finding container c6090eedba752c4c65fb5912e1fdb9ad3d891ceeadb18be21092b75dabf70213: Status 404 returned error can't find the container with id c6090eedba752c4c65fb5912e1fdb9ad3d891ceeadb18be21092b75dabf70213 Nov 02 20:38:24.720541 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.720505 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" event={"ID":"65cfe3fa-bddf-4613-8987-9ebc09c064b4","Type":"ContainerStarted","Data":"d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991"} Nov 02 20:38:24.720541 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.720548 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" event={"ID":"65cfe3fa-bddf-4613-8987-9ebc09c064b4","Type":"ContainerStarted","Data":"c6090eedba752c4c65fb5912e1fdb9ad3d891ceeadb18be21092b75dabf70213"} Nov 02 20:38:24.748153 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.748067 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:24.829917 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.829852 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" podStartSLOduration=1.8298352580000001 podStartE2EDuration="1.829835258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:24.829263231 +0000 UTC m=+2741.637047467" watchObservedRunningTime="2026-11-02 20:38:24.829835258 +0000 UTC m=+2741.637619473" Nov 02 20:38:24.923444 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:24.923412 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:25.102082 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:25.102011 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:25.812250 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:25.812218 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:30.376665 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:30.376632 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:33.779303 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:38:33.779262 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:38:34.101591 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:34.101512 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:34.114266 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:34.114241 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:34.782923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:34.782890 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:38:34.865153 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:34.865115 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:39.364654 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:39.364618 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:43.860478 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:43.860445 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:48.346591 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:48.346550 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:52.857914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:52.857862 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:56.670901 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:56.670843 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:57.351596 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:38:57.351561 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:01.863153 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:01.863115 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:03.781970 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:39:03.781930 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:39:06.395382 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:06.395350 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:10.871746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:10.871705 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:12.109479 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:12.109445 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s"] Nov 02 20:39:12.110008 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:12.109652 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" podUID="fa33f7d5-310c-4581-84c4-317226e57817" containerName="istio-proxy" containerID="cri-o://20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7" gracePeriod=30 Nov 02 20:39:12.156409 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:12.156374 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v"] Nov 02 20:39:12.156615 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:12.156579 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerName="istio-proxy" containerID="cri-o://d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1" gracePeriod=30 Nov 02 20:39:14.616613 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:14.616578 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-pbl5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:39:14.617037 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:14.616663 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" podUID="fa33f7d5-310c-4581-84c4-317226e57817" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:39:15.073473 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:15.073432 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8zc4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:39:15.073658 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:15.073497 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:39:15.342411 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:15.342328 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:17.422803 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.422771 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:39:17.443229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.443206 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:39:17.484655 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.484625 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-credential-socket\") pod \"fa33f7d5-310c-4581-84c4-317226e57817\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " Nov 02 20:39:17.484832 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.484685 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-kube-api-access-5ld52\" (UniqueName: \"kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-kube-api-access-5ld52\") pod \"c8a80651-6634-491d-89cb-59939a3c7ef8\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " Nov 02 20:39:17.484832 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.484704 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-istio-token\" (UniqueName: \"kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-istio-token\") pod \"fa33f7d5-310c-4581-84c4-317226e57817\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " Nov 02 20:39:17.484832 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.484723 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a80651-6634-491d-89cb-59939a3c7ef8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a80651-6634-491d-89cb-59939a3c7ef8-istiod-ca-cert\") pod \"c8a80651-6634-491d-89cb-59939a3c7ef8\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " Nov 02 20:39:17.484832 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.484751 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-data\") pod \"fa33f7d5-310c-4581-84c4-317226e57817\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " Nov 02 20:39:17.484832 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.484779 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa33f7d5-310c-4581-84c4-317226e57817-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa33f7d5-310c-4581-84c4-317226e57817-istio-podinfo\") pod \"fa33f7d5-310c-4581-84c4-317226e57817\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " Nov 02 20:39:17.484832 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.484821 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-token\" (UniqueName: \"kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-token\") pod \"c8a80651-6634-491d-89cb-59939a3c7ef8\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " Nov 02 20:39:17.485197 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.484862 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-data\") pod \"c8a80651-6634-491d-89cb-59939a3c7ef8\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " Nov 02 20:39:17.485197 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.484929 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-podinfo\") pod \"c8a80651-6634-491d-89cb-59939a3c7ef8\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " Nov 02 20:39:17.485197 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.484928 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-credential-socket" pod "fa33f7d5-310c-4581-84c4-317226e57817" (UID: "fa33f7d5-310c-4581-84c4-317226e57817"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:17.485197 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485064 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-certs\") pod \"fa33f7d5-310c-4581-84c4-317226e57817\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " Nov 02 20:39:17.485197 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485095 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-credential-socket\") pod \"c8a80651-6634-491d-89cb-59939a3c7ef8\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " Nov 02 20:39:17.485197 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485126 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa33f7d5-310c-4581-84c4-317226e57817-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa33f7d5-310c-4581-84c4-317226e57817-istiod-ca-cert\") pod \"fa33f7d5-310c-4581-84c4-317226e57817\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " Nov 02 20:39:17.485197 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485155 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-kube-api-access-rznlx\" (UniqueName: \"kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-kube-api-access-rznlx\") pod \"fa33f7d5-310c-4581-84c4-317226e57817\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " Nov 02 20:39:17.485197 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485189 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-envoy\") pod \"fa33f7d5-310c-4581-84c4-317226e57817\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " Nov 02 20:39:17.485622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485207 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-data" pod "c8a80651-6634-491d-89cb-59939a3c7ef8" (UID: "c8a80651-6634-491d-89cb-59939a3c7ef8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:17.485622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485225 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-envoy\") pod \"c8a80651-6634-491d-89cb-59939a3c7ef8\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " Nov 02 20:39:17.485622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485251 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-socket\") pod \"c8a80651-6634-491d-89cb-59939a3c7ef8\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " Nov 02 20:39:17.485622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485302 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-socket\") pod \"fa33f7d5-310c-4581-84c4-317226e57817\" (UID: \"fa33f7d5-310c-4581-84c4-317226e57817\") " Nov 02 20:39:17.485622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485324 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a80651-6634-491d-89cb-59939a3c7ef8-istiod-ca-cert" pod "c8a80651-6634-491d-89cb-59939a3c7ef8" (UID: "c8a80651-6634-491d-89cb-59939a3c7ef8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:17.485622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485328 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-certs\") pod \"c8a80651-6634-491d-89cb-59939a3c7ef8\" (UID: \"c8a80651-6634-491d-89cb-59939a3c7ef8\") " Nov 02 20:39:17.485622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485352 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-certs" pod "fa33f7d5-310c-4581-84c4-317226e57817" (UID: "fa33f7d5-310c-4581-84c4-317226e57817"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:17.485622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485517 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-data" pod "fa33f7d5-310c-4581-84c4-317226e57817" (UID: "fa33f7d5-310c-4581-84c4-317226e57817"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:17.486105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485729 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-credential-socket" pod "c8a80651-6634-491d-89cb-59939a3c7ef8" (UID: "c8a80651-6634-491d-89cb-59939a3c7ef8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:17.486105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485757 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa33f7d5-310c-4581-84c4-317226e57817-istiod-ca-cert" pod "fa33f7d5-310c-4581-84c4-317226e57817" (UID: "fa33f7d5-310c-4581-84c4-317226e57817"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:17.486105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485864 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-certs" pod "c8a80651-6634-491d-89cb-59939a3c7ef8" (UID: "c8a80651-6634-491d-89cb-59939a3c7ef8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:17.486105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485902 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a80651-6634-491d-89cb-59939a3c7ef8-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.486105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485913 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-socket" pod "c8a80651-6634-491d-89cb-59939a3c7ef8" (UID: "c8a80651-6634-491d-89cb-59939a3c7ef8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:17.486105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485923 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.486105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485937 5561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-data\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.486105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485952 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.486105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485968 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.486105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485982 5561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa33f7d5-310c-4581-84c4-317226e57817-istiod-ca-cert\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.486105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.485996 5561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-credential-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.486686 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.486119 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-socket" pod "fa33f7d5-310c-4581-84c4-317226e57817" (UID: "fa33f7d5-310c-4581-84c4-317226e57817"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:17.487680 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.487553 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa33f7d5-310c-4581-84c4-317226e57817-istio-podinfo" pod "fa33f7d5-310c-4581-84c4-317226e57817" (UID: "fa33f7d5-310c-4581-84c4-317226e57817"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:39:17.487680 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.487641 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-podinfo" pod "c8a80651-6634-491d-89cb-59939a3c7ef8" (UID: "c8a80651-6634-491d-89cb-59939a3c7ef8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:39:17.488026 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.487916 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-istio-token" pod "fa33f7d5-310c-4581-84c4-317226e57817" (UID: "fa33f7d5-310c-4581-84c4-317226e57817"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:17.488146 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.488122 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-kube-api-access-rznlx" pod "fa33f7d5-310c-4581-84c4-317226e57817" (UID: "fa33f7d5-310c-4581-84c4-317226e57817"). InnerVolumeSpecName "kube-api-access-rznlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:17.488263 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.488244 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-kube-api-access-5ld52" pod "c8a80651-6634-491d-89cb-59939a3c7ef8" (UID: "c8a80651-6634-491d-89cb-59939a3c7ef8"). InnerVolumeSpecName "kube-api-access-5ld52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:17.488410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.488386 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-token" pod "c8a80651-6634-491d-89cb-59939a3c7ef8" (UID: "c8a80651-6634-491d-89cb-59939a3c7ef8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:17.488410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.488395 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-envoy" pod "c8a80651-6634-491d-89cb-59939a3c7ef8" (UID: "c8a80651-6634-491d-89cb-59939a3c7ef8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:17.488587 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.488571 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-envoy" pod "fa33f7d5-310c-4581-84c4-317226e57817" (UID: "fa33f7d5-310c-4581-84c4-317226e57817"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:17.587630 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.587594 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ld52\" (UniqueName: \"kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-kube-api-access-5ld52\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.587630 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.587624 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.587630 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.587633 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa33f7d5-310c-4581-84c4-317226e57817-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.587977 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.587642 5561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-token\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.587977 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.587651 5561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-podinfo\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.587977 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.587659 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rznlx\" (UniqueName: \"kubernetes.io/projected/fa33f7d5-310c-4581-84c4-317226e57817-kube-api-access-rznlx\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.587977 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.587668 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.587977 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.587677 5561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-istio-envoy\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.587977 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.587687 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.587977 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.587695 5561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa33f7d5-310c-4581-84c4-317226e57817-workload-socket\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:17.587977 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:17.587703 5561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a80651-6634-491d-89cb-59939a3c7ef8-workload-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:18.009115 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.009074 5561 generic.go:356] "Generic (PLEG): container finished" podID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerID="d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1" exitCode=0 Nov 02 20:39:18.009115 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.009118 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" Nov 02 20:39:18.009360 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.009159 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" event={"ID":"c8a80651-6634-491d-89cb-59939a3c7ef8","Type":"ContainerDied","Data":"d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1"} Nov 02 20:39:18.009360 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.009215 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v" event={"ID":"c8a80651-6634-491d-89cb-59939a3c7ef8","Type":"ContainerDied","Data":"75a7ab75327e86ae6153723cba7f5581339d4b3b2413c76b8a8a462cb116e74b"} Nov 02 20:39:18.009360 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.009241 5561 scope.go:122] "RemoveContainer" containerID="d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1" Nov 02 20:39:18.010615 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.010590 5561 generic.go:356] "Generic (PLEG): container finished" podID="fa33f7d5-310c-4581-84c4-317226e57817" containerID="20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7" exitCode=0 Nov 02 20:39:18.010727 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.010642 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" Nov 02 20:39:18.010727 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.010672 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" event={"ID":"fa33f7d5-310c-4581-84c4-317226e57817","Type":"ContainerDied","Data":"20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7"} Nov 02 20:39:18.010727 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.010708 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s" event={"ID":"fa33f7d5-310c-4581-84c4-317226e57817","Type":"ContainerDied","Data":"107f1080b97af2cb5e9970473279820f57243e8f80f114b15c2ce5e04c516192"} Nov 02 20:39:18.029737 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.029718 5561 scope.go:122] "RemoveContainer" containerID="d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1" Nov 02 20:39:18.030065 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:39:18.030044 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1\": container with ID starting with d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1 not found: ID does not exist" containerID="d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1" Nov 02 20:39:18.030142 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.030077 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1"} err="failed to get container status \"d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1\": rpc error: code = NotFound desc = could not find container \"d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1\": container with ID starting with d6758610e013ff1bb4823e36932176c1752285822028c63b600608f3c371f5c1 not found: ID does not exist" Nov 02 20:39:18.030142 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.030103 5561 scope.go:122] "RemoveContainer" containerID="20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7" Nov 02 20:39:18.039527 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.039108 5561 scope.go:122] "RemoveContainer" containerID="20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7" Nov 02 20:39:18.040947 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:39:18.040921 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7\": container with ID starting with 20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7 not found: ID does not exist" containerID="20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7" Nov 02 20:39:18.041048 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.040955 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7"} err="failed to get container status \"20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7\": rpc error: code = NotFound desc = could not find container \"20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7\": container with ID starting with 20d5b687228ce20274b40ae5d2c2e3fbd50965058efb3b0a930cd39edae8cea7 not found: ID does not exist" Nov 02 20:39:18.165446 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.165411 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s"] Nov 02 20:39:18.212906 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.212861 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-pbl5s"] Nov 02 20:39:18.316789 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.316707 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v"] Nov 02 20:39:18.322444 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:18.322410 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8zc4v"] Nov 02 20:39:19.312507 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:19.312472 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" path="/var/lib/kubelet/pods/c8a80651-6634-491d-89cb-59939a3c7ef8/volumes" Nov 02 20:39:19.313117 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:19.313101 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa33f7d5-310c-4581-84c4-317226e57817" path="/var/lib/kubelet/pods/fa33f7d5-310c-4581-84c4-317226e57817/volumes" Nov 02 20:39:19.875278 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:19.875230 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:24.369371 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:24.369277 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:28.843154 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:28.843113 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:33.392237 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:33.392200 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:33.779551 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:39:33.779505 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:39:37.880059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:37.880023 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:42.302225 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:42.302191 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj"] Nov 02 20:39:42.359430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:42.359380 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:42.531344 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:42.531309 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:46.871430 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:46.871393 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:51.491026 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:51.490979 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:56.056889 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:56.056849 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:57.001809 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.001765 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:57.322818 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.322732 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" podUID="6faf63d9-03e2-4f8e-8a04-92d2f6903e74" containerName="main" containerID="cri-o://84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8" gracePeriod=45 Nov 02 20:39:57.519715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.519635 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:57.595491 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.595464 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:39:57.691346 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.691316 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-home\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-home\") pod \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " Nov 02 20:39:57.691520 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.691354 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-model-cache\") pod \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " Nov 02 20:39:57.691520 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.691378 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tmp-dir\") pod \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " Nov 02 20:39:57.691520 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.691422 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-dshm\") pod \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " Nov 02 20:39:57.691520 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.691464 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-kube-api-access-tt6bs\" (UniqueName: \"kubernetes.io/projected/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-kube-api-access-tt6bs\") pod \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " Nov 02 20:39:57.691520 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.691495 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tls-certs\") pod \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\" (UID: \"6faf63d9-03e2-4f8e-8a04-92d2f6903e74\") " Nov 02 20:39:57.691787 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.691637 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-home" pod "6faf63d9-03e2-4f8e-8a04-92d2f6903e74" (UID: "6faf63d9-03e2-4f8e-8a04-92d2f6903e74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:57.691787 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.691651 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-model-cache" pod "6faf63d9-03e2-4f8e-8a04-92d2f6903e74" (UID: "6faf63d9-03e2-4f8e-8a04-92d2f6903e74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:57.691787 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.691746 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tmp-dir" pod "6faf63d9-03e2-4f8e-8a04-92d2f6903e74" (UID: "6faf63d9-03e2-4f8e-8a04-92d2f6903e74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:57.692053 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.691959 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:57.692053 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.691972 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:57.692053 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.691985 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:57.693697 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.693666 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-dshm" pod "6faf63d9-03e2-4f8e-8a04-92d2f6903e74" (UID: "6faf63d9-03e2-4f8e-8a04-92d2f6903e74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:57.693821 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.693805 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tls-certs" pod "6faf63d9-03e2-4f8e-8a04-92d2f6903e74" (UID: "6faf63d9-03e2-4f8e-8a04-92d2f6903e74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:39:57.693821 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.693807 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-kube-api-access-tt6bs" pod "6faf63d9-03e2-4f8e-8a04-92d2f6903e74" (UID: "6faf63d9-03e2-4f8e-8a04-92d2f6903e74"). InnerVolumeSpecName "kube-api-access-tt6bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:57.793433 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.793401 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:57.793433 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.793433 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt6bs\" (UniqueName: \"kubernetes.io/projected/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-kube-api-access-tt6bs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:57.793638 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:57.793449 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf63d9-03e2-4f8e-8a04-92d2f6903e74-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.223982 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:58.223940 5561 generic.go:356] "Generic (PLEG): container finished" podID="6faf63d9-03e2-4f8e-8a04-92d2f6903e74" containerID="84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8" exitCode=0 Nov 02 20:39:58.224167 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:58.224020 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" Nov 02 20:39:58.224167 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:58.224036 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" event={"ID":"6faf63d9-03e2-4f8e-8a04-92d2f6903e74","Type":"ContainerDied","Data":"84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8"} Nov 02 20:39:58.224167 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:58.224083 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj" event={"ID":"6faf63d9-03e2-4f8e-8a04-92d2f6903e74","Type":"ContainerDied","Data":"e541b1dc3b2eaadf4e5f33e7c1305d8993959af4b8e837d271e3f43fc7dcb1d5"} Nov 02 20:39:58.224167 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:58.224105 5561 scope.go:122] "RemoveContainer" containerID="84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8" Nov 02 20:39:58.240666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:58.240646 5561 scope.go:122] "RemoveContainer" containerID="84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8" Nov 02 20:39:58.241069 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:39:58.241043 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8\": container with ID starting with 84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8 not found: ID does not exist" containerID="84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8" Nov 02 20:39:58.241140 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:58.241084 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8"} err="failed to get container status \"84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8\": rpc error: code = NotFound desc = could not find container \"84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8\": container with ID starting with 84241ae4bce86d7956de2466543c113cd2b2007a52b52d54d6d8548bd97a9bd8 not found: ID does not exist" Nov 02 20:39:58.379422 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:58.379389 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj"] Nov 02 20:39:58.427005 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:58.426972 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-65f28f/test-model-65f28f-kserve-5c75898bf-rlzhj"] Nov 02 20:39:59.314670 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:39:59.314636 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6faf63d9-03e2-4f8e-8a04-92d2f6903e74" path="/var/lib/kubelet/pods/6faf63d9-03e2-4f8e-8a04-92d2f6903e74/volumes" Nov 02 20:40:00.560566 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:00.560532 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:03.780448 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:40:03.780409 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:40:04.846725 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:04.846669 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:05.195453 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:05.195421 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:10.097076 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:10.097042 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:14.645325 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:14.645289 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:17.248862 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:17.248782 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:17.420994 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:17.420947 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:19.182606 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:19.182565 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:23.670802 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:23.670764 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:28.248684 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:28.248651 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:32.779753 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:32.779716 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:33.778455 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:40:33.778418 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:40:37.252549 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:37.252511 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:41.128975 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:41.128930 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:41.777774 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:41.777737 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:46.251727 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:46.251691 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:49.337183 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:49.337144 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:50.248669 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:50.248634 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:51.668731 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:51.668693 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:51.855247 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:51.855211 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:52.029120 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:52.029037 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:52.643810 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:52.643776 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:56.664425 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:56.664339 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:57.346040 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:57.346004 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:58.712464 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:58.712433 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:58.898963 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:58.898931 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:59.072863 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:40:59.072779 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:01.457445 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:01.457408 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:03.779280 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:41:03.779229 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:41:06.279559 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:06.279521 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:10.842894 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:10.842838 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:11.369446 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:11.369412 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll"] Nov 02 20:41:11.483678 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:11.483638 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg"] Nov 02 20:41:11.787004 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:11.786959 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:11.963029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:11.962992 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:15.251931 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:15.251868 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:16.369990 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.369941 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" podUID="1797345c-e27c-423d-8b0c-71270e96647f" containerName="payload-processing" containerID="cri-o://23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de" gracePeriod=25 Nov 02 20:41:16.484040 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.483974 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" podUID="b90f9ab7-7575-4bd8-b55b-93872c815747" containerName="payload-pre-processing" containerID="cri-o://7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a" gracePeriod=25 Nov 02 20:41:16.626411 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.626349 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" Nov 02 20:41:16.647426 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.647390 5561 generic.go:356] "Generic (PLEG): container finished" podID="1797345c-e27c-423d-8b0c-71270e96647f" containerID="23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de" exitCode=0 Nov 02 20:41:16.647569 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.647427 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" Nov 02 20:41:16.647569 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.647469 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" event={"ID":"1797345c-e27c-423d-8b0c-71270e96647f","Type":"ContainerDied","Data":"23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de"} Nov 02 20:41:16.647569 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.647525 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll" event={"ID":"1797345c-e27c-423d-8b0c-71270e96647f","Type":"ContainerDied","Data":"9b69929fff8b6631cd0b8b19ea1675238d5c016d497598d73d22a3a50c190d11"} Nov 02 20:41:16.647569 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.647549 5561 scope.go:122] "RemoveContainer" containerID="23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de" Nov 02 20:41:16.656486 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.656466 5561 scope.go:122] "RemoveContainer" containerID="23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de" Nov 02 20:41:16.656766 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:41:16.656748 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de\": container with ID starting with 23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de not found: ID does not exist" containerID="23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de" Nov 02 20:41:16.656819 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.656777 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de"} err="failed to get container status \"23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de\": rpc error: code = NotFound desc = could not find container \"23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de\": container with ID starting with 23d3a080f976999f10adcb9f6523d6df2288a9a89199d7e4c8a39f976621b8de not found: ID does not exist" Nov 02 20:41:16.691344 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.691318 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1797345c-e27c-423d-8b0c-71270e96647f-kube-api-access-9dl6n\" (UniqueName: \"kubernetes.io/projected/1797345c-e27c-423d-8b0c-71270e96647f-kube-api-access-9dl6n\") pod \"1797345c-e27c-423d-8b0c-71270e96647f\" (UID: \"1797345c-e27c-423d-8b0c-71270e96647f\") " Nov 02 20:41:16.691487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.691471 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1797345c-e27c-423d-8b0c-71270e96647f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797345c-e27c-423d-8b0c-71270e96647f-plugins-config-volume\") pod \"1797345c-e27c-423d-8b0c-71270e96647f\" (UID: \"1797345c-e27c-423d-8b0c-71270e96647f\") " Nov 02 20:41:16.691857 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.691835 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1797345c-e27c-423d-8b0c-71270e96647f-plugins-config-volume" pod "1797345c-e27c-423d-8b0c-71270e96647f" (UID: "1797345c-e27c-423d-8b0c-71270e96647f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:41:16.693428 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.693408 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1797345c-e27c-423d-8b0c-71270e96647f-kube-api-access-9dl6n" pod "1797345c-e27c-423d-8b0c-71270e96647f" (UID: "1797345c-e27c-423d-8b0c-71270e96647f"). InnerVolumeSpecName "kube-api-access-9dl6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:41:16.801253 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.801218 5561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797345c-e27c-423d-8b0c-71270e96647f-plugins-config-volume\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:16.801253 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:16.801247 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dl6n\" (UniqueName: \"kubernetes.io/projected/1797345c-e27c-423d-8b0c-71270e96647f-kube-api-access-9dl6n\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:17.061353 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:17.061318 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll"] Nov 02 20:41:17.064362 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:17.064334 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7b1839-7b7bccbfd7-7fnll"] Nov 02 20:41:17.311945 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:17.311848 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1797345c-e27c-423d-8b0c-71270e96647f" path="/var/lib/kubelet/pods/1797345c-e27c-423d-8b0c-71270e96647f/volumes" Nov 02 20:41:19.814531 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:19.814481 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:19.923692 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:19.923654 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" podUID="b90f9ab7-7575-4bd8-b55b-93872c815747" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:41:24.344630 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:24.344589 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:28.858243 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:28.858210 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:29.923959 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:29.923915 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" podUID="b90f9ab7-7575-4bd8-b55b-93872c815747" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:41:33.379793 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:33.379757 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:33.780162 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:41:33.780114 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:41:36.727582 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.727558 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" Nov 02 20:41:36.753658 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.753582 5561 generic.go:356] "Generic (PLEG): container finished" podID="b90f9ab7-7575-4bd8-b55b-93872c815747" containerID="7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a" exitCode=0 Nov 02 20:41:36.753658 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.753616 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" event={"ID":"b90f9ab7-7575-4bd8-b55b-93872c815747","Type":"ContainerDied","Data":"7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a"} Nov 02 20:41:36.753658 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.753632 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" Nov 02 20:41:36.753910 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.753674 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg" event={"ID":"b90f9ab7-7575-4bd8-b55b-93872c815747","Type":"ContainerDied","Data":"da6c1d2ba1071f4b060277bed7d0e09dc78d6766a22e63262dde566aa82e18f1"} Nov 02 20:41:36.753910 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.753697 5561 scope.go:122] "RemoveContainer" containerID="7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a" Nov 02 20:41:36.762332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.762315 5561 scope.go:122] "RemoveContainer" containerID="7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a" Nov 02 20:41:36.762615 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:41:36.762594 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a\": container with ID starting with 7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a not found: ID does not exist" containerID="7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a" Nov 02 20:41:36.762706 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.762622 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a"} err="failed to get container status \"7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a\": rpc error: code = NotFound desc = could not find container \"7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a\": container with ID starting with 7af2f463a0800f390f17849ba87b63f09d66a63d88eb2e7aba0c4ad182a4c25a not found: ID does not exist" Nov 02 20:41:36.805341 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.805304 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b90f9ab7-7575-4bd8-b55b-93872c815747-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90f9ab7-7575-4bd8-b55b-93872c815747-plugins-config-volume\") pod \"b90f9ab7-7575-4bd8-b55b-93872c815747\" (UID: \"b90f9ab7-7575-4bd8-b55b-93872c815747\") " Nov 02 20:41:36.805475 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.805417 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90f9ab7-7575-4bd8-b55b-93872c815747-kube-api-access-9d9zn\" (UniqueName: \"kubernetes.io/projected/b90f9ab7-7575-4bd8-b55b-93872c815747-kube-api-access-9d9zn\") pod \"b90f9ab7-7575-4bd8-b55b-93872c815747\" (UID: \"b90f9ab7-7575-4bd8-b55b-93872c815747\") " Nov 02 20:41:36.805747 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.805720 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90f9ab7-7575-4bd8-b55b-93872c815747-plugins-config-volume" pod "b90f9ab7-7575-4bd8-b55b-93872c815747" (UID: "b90f9ab7-7575-4bd8-b55b-93872c815747"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:41:36.805845 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.805830 5561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90f9ab7-7575-4bd8-b55b-93872c815747-plugins-config-volume\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:36.807685 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.807665 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90f9ab7-7575-4bd8-b55b-93872c815747-kube-api-access-9d9zn" pod "b90f9ab7-7575-4bd8-b55b-93872c815747" (UID: "b90f9ab7-7575-4bd8-b55b-93872c815747"). InnerVolumeSpecName "kube-api-access-9d9zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:41:36.907413 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:36.907379 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d9zn\" (UniqueName: \"kubernetes.io/projected/b90f9ab7-7575-4bd8-b55b-93872c815747-kube-api-access-9d9zn\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:37.167436 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:37.167401 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg"] Nov 02 20:41:37.169100 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:37.169080 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7b1839-674756589f-859jg"] Nov 02 20:41:37.312299 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:37.312210 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b90f9ab7-7575-4bd8-b55b-93872c815747" path="/var/lib/kubelet/pods/b90f9ab7-7575-4bd8-b55b-93872c815747/volumes" Nov 02 20:41:37.872965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:37.872928 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:41.798514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:41.798475 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:43.211057 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:43.211025 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:43.468576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:43.468497 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:43.649921 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:43.649889 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:44.068812 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:44.068772 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:47.129965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:47.129936 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s"] Nov 02 20:41:47.189708 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:47.189675 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:48.265755 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:48.265719 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:49.842599 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:49.842553 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:50.038526 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:50.038495 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:52.517862 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:52.517825 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:53.004452 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:53.004425 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:53.178514 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:53.178482 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:55.530815 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:55.530780 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r"] Nov 02 20:41:55.582249 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:55.582204 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8"] Nov 02 20:41:55.584621 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:55.584591 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh"] Nov 02 20:41:55.702122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:55.702082 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:57.677254 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:57.677220 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:57.849676 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:41:57.849643 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:00.037413 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:00.037382 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj"] Nov 02 20:42:01.146007 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:01.145967 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:02.154017 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.153976 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" podUID="20cd41d9-5f2d-4ece-a7f7-e62487149e0a" containerName="main" containerID="cri-o://0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728" gracePeriod=45 Nov 02 20:42:02.223899 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.223834 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:02.393574 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.393542 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:02.407599 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.407543 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:42:02.465255 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.465219 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-model-cache\") pod \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " Nov 02 20:42:02.465448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.465274 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-kube-api-access-k7lz4\" (UniqueName: \"kubernetes.io/projected/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-kube-api-access-k7lz4\") pod \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " Nov 02 20:42:02.465448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.465387 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tls-certs\") pod \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " Nov 02 20:42:02.465577 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.465460 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-dshm\") pod \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " Nov 02 20:42:02.465577 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.465504 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-home\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-home\") pod \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " Nov 02 20:42:02.465577 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.465541 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tmp-dir\") pod \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\" (UID: \"20cd41d9-5f2d-4ece-a7f7-e62487149e0a\") " Nov 02 20:42:02.465577 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.465548 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-model-cache" pod "20cd41d9-5f2d-4ece-a7f7-e62487149e0a" (UID: "20cd41d9-5f2d-4ece-a7f7-e62487149e0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:02.465829 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.465796 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tmp-dir" pod "20cd41d9-5f2d-4ece-a7f7-e62487149e0a" (UID: "20cd41d9-5f2d-4ece-a7f7-e62487149e0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:02.465829 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.465804 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-home" pod "20cd41d9-5f2d-4ece-a7f7-e62487149e0a" (UID: "20cd41d9-5f2d-4ece-a7f7-e62487149e0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:02.465990 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.465871 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:02.467733 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.467707 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-dshm" pod "20cd41d9-5f2d-4ece-a7f7-e62487149e0a" (UID: "20cd41d9-5f2d-4ece-a7f7-e62487149e0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:02.467828 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.467769 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tls-certs" pod "20cd41d9-5f2d-4ece-a7f7-e62487149e0a" (UID: "20cd41d9-5f2d-4ece-a7f7-e62487149e0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:02.467828 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.467796 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-kube-api-access-k7lz4" pod "20cd41d9-5f2d-4ece-a7f7-e62487149e0a" (UID: "20cd41d9-5f2d-4ece-a7f7-e62487149e0a"). InnerVolumeSpecName "kube-api-access-k7lz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:02.567395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.567348 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7lz4\" (UniqueName: \"kubernetes.io/projected/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-kube-api-access-k7lz4\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:02.567395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.567386 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:02.567395 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.567399 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:02.567646 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.567410 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:02.567646 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.567435 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cd41d9-5f2d-4ece-a7f7-e62487149e0a-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:02.879470 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.879378 5561 generic.go:356] "Generic (PLEG): container finished" podID="20cd41d9-5f2d-4ece-a7f7-e62487149e0a" containerID="0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728" exitCode=0 Nov 02 20:42:02.879470 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.879432 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" Nov 02 20:42:02.879470 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.879460 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" event={"ID":"20cd41d9-5f2d-4ece-a7f7-e62487149e0a","Type":"ContainerDied","Data":"0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728"} Nov 02 20:42:02.879720 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.879504 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s" event={"ID":"20cd41d9-5f2d-4ece-a7f7-e62487149e0a","Type":"ContainerDied","Data":"dd23a50145e69ce8f0031978359bb88ecb7f27ea9fc4fd9bdea8a2531cdcd7d6"} Nov 02 20:42:02.879720 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.879521 5561 scope.go:122] "RemoveContainer" containerID="0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728" Nov 02 20:42:02.889037 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.889017 5561 scope.go:122] "RemoveContainer" containerID="0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728" Nov 02 20:42:02.889372 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:42:02.889351 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728\": container with ID starting with 0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728 not found: ID does not exist" containerID="0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728" Nov 02 20:42:02.889411 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:02.889383 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728"} err="failed to get container status \"0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728\": rpc error: code = NotFound desc = could not find container \"0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728\": container with ID starting with 0d77bfc070d153702111e1363ae5dc37deffb83556e268605398ecc78cb78728 not found: ID does not exist" Nov 02 20:42:03.034836 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:03.034803 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s"] Nov 02 20:42:03.082356 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:03.082317 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0d9ec0/test-model-0d9ec0-kserve-5c75f5bf-kdq8s"] Nov 02 20:42:03.312670 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:03.312637 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20cd41d9-5f2d-4ece-a7f7-e62487149e0a" path="/var/lib/kubelet/pods/20cd41d9-5f2d-4ece-a7f7-e62487149e0a/volumes" Nov 02 20:42:03.781435 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:42:03.781398 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:42:06.970181 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:06.970146 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:10.551973 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.551926 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" containerID="cri-o://7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47" gracePeriod=45 Nov 02 20:42:10.607358 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.607310 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" containerID="cri-o://cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b" gracePeriod=45 Nov 02 20:42:10.608848 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.608817 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" containerID="cri-o://ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18" gracePeriod=45 Nov 02 20:42:10.805055 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.804981 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": dial tcp 10.134.0.42:8000: connect: connection refused" Nov 02 20:42:10.880476 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.880444 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:42:10.911372 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.911341 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:42:10.915017 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.914989 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:42:10.925252 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.925211 5561 generic.go:356] "Generic (PLEG): container finished" podID="08b219f6-3577-4242-9bca-448a02c6b524" containerID="ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18" exitCode=0 Nov 02 20:42:10.925446 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.925320 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" Nov 02 20:42:10.925446 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.925322 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" event={"ID":"08b219f6-3577-4242-9bca-448a02c6b524","Type":"ContainerDied","Data":"ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18"} Nov 02 20:42:10.925446 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.925435 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh" event={"ID":"08b219f6-3577-4242-9bca-448a02c6b524","Type":"ContainerDied","Data":"3aa4cf6d3d4afb15093644108af1ce14062ed5fc6de6e2038b319f1df218f972"} Nov 02 20:42:10.925586 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.925467 5561 scope.go:122] "RemoveContainer" containerID="ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18" Nov 02 20:42:10.926732 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.926698 5561 generic.go:356] "Generic (PLEG): container finished" podID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerID="7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47" exitCode=0 Nov 02 20:42:10.926858 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.926727 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" event={"ID":"51e6bd1e-cd4e-45a4-a9da-9003f53f359e","Type":"ContainerDied","Data":"7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47"} Nov 02 20:42:10.926858 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.926749 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" Nov 02 20:42:10.926858 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.926767 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r" event={"ID":"51e6bd1e-cd4e-45a4-a9da-9003f53f359e","Type":"ContainerDied","Data":"e2def228f2c0b2f32988fb34eed5c95fac364a5811debde8698e155b14a7c996"} Nov 02 20:42:10.928050 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.928030 5561 generic.go:356] "Generic (PLEG): container finished" podID="41b6de3b-682d-4a07-9297-e67df9f94243" containerID="cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b" exitCode=0 Nov 02 20:42:10.928150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.928102 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" Nov 02 20:42:10.928150 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.928125 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" event={"ID":"41b6de3b-682d-4a07-9297-e67df9f94243","Type":"ContainerDied","Data":"cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b"} Nov 02 20:42:10.928295 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.928152 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8" event={"ID":"41b6de3b-682d-4a07-9297-e67df9f94243","Type":"ContainerDied","Data":"d8cdae0b70e88bf2c440bd06fa1ece73ab5853f864b9107cdc00e990206d27f6"} Nov 02 20:42:10.935381 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.935358 5561 scope.go:122] "RemoveContainer" containerID="ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18" Nov 02 20:42:10.935697 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:42:10.935680 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18\": container with ID starting with ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18 not found: ID does not exist" containerID="ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18" Nov 02 20:42:10.935745 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.935707 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18"} err="failed to get container status \"ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18\": rpc error: code = NotFound desc = could not find container \"ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18\": container with ID starting with ab0a2404f27508e28753132f374a2e0e1082ecdef954ca260c141dd524c36b18 not found: ID does not exist" Nov 02 20:42:10.935745 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.935726 5561 scope.go:122] "RemoveContainer" containerID="7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47" Nov 02 20:42:10.946321 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.946296 5561 scope.go:122] "RemoveContainer" containerID="7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47" Nov 02 20:42:10.946675 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:42:10.946649 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47\": container with ID starting with 7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47 not found: ID does not exist" containerID="7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47" Nov 02 20:42:10.946771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.946686 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47"} err="failed to get container status \"7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47\": rpc error: code = NotFound desc = could not find container \"7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47\": container with ID starting with 7348cae8de43b678a82406d312a341d04aeb5339425629b743f9969cb763cb47 not found: ID does not exist" Nov 02 20:42:10.946771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.946707 5561 scope.go:122] "RemoveContainer" containerID="cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b" Nov 02 20:42:10.952492 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.952463 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tmp-dir\") pod \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " Nov 02 20:42:10.952590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.952514 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-kube-api-access-xbwdb\" (UniqueName: \"kubernetes.io/projected/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-kube-api-access-xbwdb\") pod \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " Nov 02 20:42:10.952590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.952581 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-model-cache\") pod \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " Nov 02 20:42:10.952696 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.952668 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-dshm\") pod \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " Nov 02 20:42:10.952734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.952703 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tls-certs\" (UniqueName: \"kubernetes.io/secret/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tls-certs\") pod \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " Nov 02 20:42:10.952781 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.952766 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-home\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-home\") pod \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\" (UID: \"51e6bd1e-cd4e-45a4-a9da-9003f53f359e\") " Nov 02 20:42:10.953339 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.953315 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-home" pod "51e6bd1e-cd4e-45a4-a9da-9003f53f359e" (UID: "51e6bd1e-cd4e-45a4-a9da-9003f53f359e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:10.953535 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.953512 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tmp-dir" pod "51e6bd1e-cd4e-45a4-a9da-9003f53f359e" (UID: "51e6bd1e-cd4e-45a4-a9da-9003f53f359e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:10.954194 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.954169 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-model-cache" pod "51e6bd1e-cd4e-45a4-a9da-9003f53f359e" (UID: "51e6bd1e-cd4e-45a4-a9da-9003f53f359e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:10.956103 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.956079 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-dshm" pod "51e6bd1e-cd4e-45a4-a9da-9003f53f359e" (UID: "51e6bd1e-cd4e-45a4-a9da-9003f53f359e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:10.956232 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.956142 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-kube-api-access-xbwdb" pod "51e6bd1e-cd4e-45a4-a9da-9003f53f359e" (UID: "51e6bd1e-cd4e-45a4-a9da-9003f53f359e"). InnerVolumeSpecName "kube-api-access-xbwdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:10.956473 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.956449 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tls-certs" pod "51e6bd1e-cd4e-45a4-a9da-9003f53f359e" (UID: "51e6bd1e-cd4e-45a4-a9da-9003f53f359e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:10.960785 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.960768 5561 scope.go:122] "RemoveContainer" containerID="cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b" Nov 02 20:42:10.961385 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:42:10.961359 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b\": container with ID starting with cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b not found: ID does not exist" containerID="cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b" Nov 02 20:42:10.961463 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:10.961398 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b"} err="failed to get container status \"cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b\": rpc error: code = NotFound desc = could not find container \"cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b\": container with ID starting with cb5eeaa6ebd2dac106f49306f39cf29be1829100a53e643d28cd24afabb8b67b not found: ID does not exist" Nov 02 20:42:11.008489 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.008448 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:11.054290 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054259 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-dshm\") pod \"41b6de3b-682d-4a07-9297-e67df9f94243\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " Nov 02 20:42:11.054452 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054338 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-home\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-home\") pod \"41b6de3b-682d-4a07-9297-e67df9f94243\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " Nov 02 20:42:11.054452 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054377 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b6de3b-682d-4a07-9297-e67df9f94243-kube-api-access-cw2qb\" (UniqueName: \"kubernetes.io/projected/41b6de3b-682d-4a07-9297-e67df9f94243-kube-api-access-cw2qb\") pod \"41b6de3b-682d-4a07-9297-e67df9f94243\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " Nov 02 20:42:11.054452 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054412 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-dshm\") pod \"08b219f6-3577-4242-9bca-448a02c6b524\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " Nov 02 20:42:11.054452 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054429 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-home\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-home\") pod \"08b219f6-3577-4242-9bca-448a02c6b524\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " Nov 02 20:42:11.054658 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054481 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08b219f6-3577-4242-9bca-448a02c6b524-tls-certs\" (UniqueName: \"kubernetes.io/secret/08b219f6-3577-4242-9bca-448a02c6b524-tls-certs\") pod \"08b219f6-3577-4242-9bca-448a02c6b524\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " Nov 02 20:42:11.054658 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054618 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-home" pod "41b6de3b-682d-4a07-9297-e67df9f94243" (UID: "41b6de3b-682d-4a07-9297-e67df9f94243"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:11.054658 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054635 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41b6de3b-682d-4a07-9297-e67df9f94243-tls-certs\" (UniqueName: \"kubernetes.io/secret/41b6de3b-682d-4a07-9297-e67df9f94243-tls-certs\") pod \"41b6de3b-682d-4a07-9297-e67df9f94243\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " Nov 02 20:42:11.054820 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054691 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-model-cache\") pod \"41b6de3b-682d-4a07-9297-e67df9f94243\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " Nov 02 20:42:11.054820 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054735 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b219f6-3577-4242-9bca-448a02c6b524-kube-api-access-2km4l\" (UniqueName: \"kubernetes.io/projected/08b219f6-3577-4242-9bca-448a02c6b524-kube-api-access-2km4l\") pod \"08b219f6-3577-4242-9bca-448a02c6b524\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " Nov 02 20:42:11.054820 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054777 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-tmp-dir\") pod \"41b6de3b-682d-4a07-9297-e67df9f94243\" (UID: \"41b6de3b-682d-4a07-9297-e67df9f94243\") " Nov 02 20:42:11.055030 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054899 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-model-cache\") pod \"08b219f6-3577-4242-9bca-448a02c6b524\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " Nov 02 20:42:11.055030 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.054943 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-tmp-dir\") pod \"08b219f6-3577-4242-9bca-448a02c6b524\" (UID: \"08b219f6-3577-4242-9bca-448a02c6b524\") " Nov 02 20:42:11.055172 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.055026 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-home" pod "08b219f6-3577-4242-9bca-448a02c6b524" (UID: "08b219f6-3577-4242-9bca-448a02c6b524"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:11.055342 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.055318 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.055420 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.055349 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.055420 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.055391 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.055420 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.055407 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.055580 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.055422 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.055580 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.055435 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.055580 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.055447 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.055580 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.055462 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbwdb\" (UniqueName: \"kubernetes.io/projected/51e6bd1e-cd4e-45a4-a9da-9003f53f359e-kube-api-access-xbwdb\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.055790 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.055679 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-tmp-dir" pod "08b219f6-3577-4242-9bca-448a02c6b524" (UID: "08b219f6-3577-4242-9bca-448a02c6b524"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:11.055900 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.055857 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-tmp-dir" pod "41b6de3b-682d-4a07-9297-e67df9f94243" (UID: "41b6de3b-682d-4a07-9297-e67df9f94243"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:11.056163 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.056139 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-model-cache" pod "08b219f6-3577-4242-9bca-448a02c6b524" (UID: "08b219f6-3577-4242-9bca-448a02c6b524"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:11.056388 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.056365 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-model-cache" pod "41b6de3b-682d-4a07-9297-e67df9f94243" (UID: "41b6de3b-682d-4a07-9297-e67df9f94243"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:11.056997 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.056975 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b219f6-3577-4242-9bca-448a02c6b524-tls-certs" pod "08b219f6-3577-4242-9bca-448a02c6b524" (UID: "08b219f6-3577-4242-9bca-448a02c6b524"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:11.057229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.057010 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-dshm" pod "41b6de3b-682d-4a07-9297-e67df9f94243" (UID: "41b6de3b-682d-4a07-9297-e67df9f94243"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:11.057229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.057011 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b6de3b-682d-4a07-9297-e67df9f94243-kube-api-access-cw2qb" pod "41b6de3b-682d-4a07-9297-e67df9f94243" (UID: "41b6de3b-682d-4a07-9297-e67df9f94243"). InnerVolumeSpecName "kube-api-access-cw2qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:11.057340 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.057073 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41b6de3b-682d-4a07-9297-e67df9f94243-tls-certs" pod "41b6de3b-682d-4a07-9297-e67df9f94243" (UID: "41b6de3b-682d-4a07-9297-e67df9f94243"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:11.057388 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.057370 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b219f6-3577-4242-9bca-448a02c6b524-kube-api-access-2km4l" pod "08b219f6-3577-4242-9bca-448a02c6b524" (UID: "08b219f6-3577-4242-9bca-448a02c6b524"). InnerVolumeSpecName "kube-api-access-2km4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:11.058174 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.058144 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-dshm" pod "08b219f6-3577-4242-9bca-448a02c6b524" (UID: "08b219f6-3577-4242-9bca-448a02c6b524"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:11.160730 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.160694 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.160730 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.160738 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08b219f6-3577-4242-9bca-448a02c6b524-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.160999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.160753 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41b6de3b-682d-4a07-9297-e67df9f94243-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.160999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.160770 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.160999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.160787 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2km4l\" (UniqueName: \"kubernetes.io/projected/08b219f6-3577-4242-9bca-448a02c6b524-kube-api-access-2km4l\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.160999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.160806 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.160999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.160819 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.160999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.160835 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08b219f6-3577-4242-9bca-448a02c6b524-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.160999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.160848 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b6de3b-682d-4a07-9297-e67df9f94243-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.160999 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.160865 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw2qb\" (UniqueName: \"kubernetes.io/projected/41b6de3b-682d-4a07-9297-e67df9f94243-kube-api-access-cw2qb\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.359965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.359855 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh"] Nov 02 20:42:11.365033 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.364999 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/simulator-w5-ae8e6e-kserve-6547b44896-jkzmh"] Nov 02 20:42:11.455630 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.455595 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:11.468016 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.467980 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r"] Nov 02 20:42:11.471505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.471477 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/distinct-2-w5-ae8e6e-kserve-7945b46f79-qfk2r"] Nov 02 20:42:11.574952 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.574923 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8"] Nov 02 20:42:11.580037 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.580006 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/distinct-w5-ae8e6e-kserve-64594bdd96-49hq8"] Nov 02 20:42:11.733078 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:11.733047 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:13.312848 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:13.312813 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b219f6-3577-4242-9bca-448a02c6b524" path="/var/lib/kubelet/pods/08b219f6-3577-4242-9bca-448a02c6b524/volumes" Nov 02 20:42:13.313434 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:13.313413 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" path="/var/lib/kubelet/pods/41b6de3b-682d-4a07-9297-e67df9f94243/volumes" Nov 02 20:42:13.313852 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:13.313832 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" path="/var/lib/kubelet/pods/51e6bd1e-cd4e-45a4-a9da-9003f53f359e/volumes" Nov 02 20:42:15.059126 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.059068 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" podUID="8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" containerName="main" containerID="cri-o://994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1" gracePeriod=45 Nov 02 20:42:15.197217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.197161 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" podUID="8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": dial tcp 10.134.0.97:8000: connect: connection refused" Nov 02 20:42:15.284050 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.283966 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:15.345463 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.345435 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:42:15.458629 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.458591 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:15.501394 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.501358 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-home\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-home\") pod \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " Nov 02 20:42:15.501590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.501424 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tls-certs\") pod \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " Nov 02 20:42:15.501590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.501447 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-kube-api-access-68w9g\" (UniqueName: \"kubernetes.io/projected/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-kube-api-access-68w9g\") pod \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " Nov 02 20:42:15.501590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.501477 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-model-cache\") pod \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " Nov 02 20:42:15.501590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.501511 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-dshm\") pod \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " Nov 02 20:42:15.501590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.501560 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tmp-dir\") pod \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\" (UID: \"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1\") " Nov 02 20:42:15.501895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.501709 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-home" pod "8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" (UID: "8bf23a0b-8cef-49dc-87f8-68ad4f0547b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:15.501895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.501793 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-model-cache" pod "8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" (UID: "8bf23a0b-8cef-49dc-87f8-68ad4f0547b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:15.502017 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.501916 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:15.502017 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.501934 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:15.502127 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.502026 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tmp-dir" pod "8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" (UID: "8bf23a0b-8cef-49dc-87f8-68ad4f0547b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:15.503846 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.503813 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-kube-api-access-68w9g" pod "8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" (UID: "8bf23a0b-8cef-49dc-87f8-68ad4f0547b1"). InnerVolumeSpecName "kube-api-access-68w9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:15.504095 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.504067 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-dshm" pod "8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" (UID: "8bf23a0b-8cef-49dc-87f8-68ad4f0547b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:15.504095 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.504085 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tls-certs" pod "8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" (UID: "8bf23a0b-8cef-49dc-87f8-68ad4f0547b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:15.603095 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.603012 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:15.603095 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.603045 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68w9g\" (UniqueName: \"kubernetes.io/projected/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-kube-api-access-68w9g\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:15.603095 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.603059 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:15.603095 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.603096 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:15.957836 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.957791 5561 generic.go:356] "Generic (PLEG): container finished" podID="8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" containerID="994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1" exitCode=0 Nov 02 20:42:15.958059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.957929 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" event={"ID":"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1","Type":"ContainerDied","Data":"994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1"} Nov 02 20:42:15.958129 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.958043 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" event={"ID":"8bf23a0b-8cef-49dc-87f8-68ad4f0547b1","Type":"ContainerDied","Data":"d704ef3e9d4d5fd62c1362a2106211506751701a297755d757e6d46c5a49dc4e"} Nov 02 20:42:15.958129 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.958099 5561 scope.go:122] "RemoveContainer" containerID="994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1" Nov 02 20:42:15.958296 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.958228 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj" Nov 02 20:42:15.958495 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.958472 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:15.967897 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.967862 5561 scope.go:122] "RemoveContainer" containerID="994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1" Nov 02 20:42:15.968189 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:42:15.968172 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1\": container with ID starting with 994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1 not found: ID does not exist" containerID="994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1" Nov 02 20:42:15.968242 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:15.968197 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1"} err="failed to get container status \"994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1\": rpc error: code = NotFound desc = could not find container \"994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1\": container with ID starting with 994ad7aaae14a3c2159f50475da67168c92d4f6088e284ac34cd8d2b89b129d1 not found: ID does not exist" Nov 02 20:42:16.112890 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:16.112843 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj"] Nov 02 20:42:16.118047 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:16.118023 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bb78ba/test-model-bb78ba-kserve-5588665c75-m7mbj"] Nov 02 20:42:16.683422 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:16.683389 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:17.314651 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:17.314618 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" path="/var/lib/kubelet/pods/8bf23a0b-8cef-49dc-87f8-68ad4f0547b1/volumes" Nov 02 20:42:20.571824 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:20.571787 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:22.390125 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:22.390088 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:25.063147 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:25.063066 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:27.039070 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:27.039030 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78"] Nov 02 20:42:27.093388 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:27.093348 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:29.702083 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:29.702046 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:33.778325 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:42:33.778288 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:42:34.800315 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:34.800277 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:36.627221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:36.627188 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:38.571804 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:38.571770 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p"] Nov 02 20:42:38.640387 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:38.640353 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:39.261001 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:39.260962 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:39.742056 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:39.742004 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:39.918858 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:39.918800 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:40.382431 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.382397 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh"] Nov 02 20:42:40.382617 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.382577 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" podUID="c94eb3e8-cfd0-46ca-8b24-6746f4437e4e" containerName="maas-api" containerID="cri-o://7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291" gracePeriod=30 Nov 02 20:42:40.534459 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.534385 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:40.657055 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.657028 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:42:40.750911 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.750854 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-openshift-service-ca\") pod \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " Nov 02 20:42:40.751302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.750972 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-kube-api-access-k8qtp\" (UniqueName: \"kubernetes.io/projected/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-kube-api-access-k8qtp\") pod \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " Nov 02 20:42:40.751302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.751005 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-maas-api-tls\") pod \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " Nov 02 20:42:40.751302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.751055 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-metrics-certs\") pod \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\" (UID: \"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e\") " Nov 02 20:42:40.751302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.751253 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-openshift-service-ca" pod "c94eb3e8-cfd0-46ca-8b24-6746f4437e4e" (UID: "c94eb3e8-cfd0-46ca-8b24-6746f4437e4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:40.753189 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.753159 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-maas-api-tls" pod "c94eb3e8-cfd0-46ca-8b24-6746f4437e4e" (UID: "c94eb3e8-cfd0-46ca-8b24-6746f4437e4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:40.753189 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.753174 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-metrics-certs" pod "c94eb3e8-cfd0-46ca-8b24-6746f4437e4e" (UID: "c94eb3e8-cfd0-46ca-8b24-6746f4437e4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:40.753344 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.753214 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-kube-api-access-k8qtp" pod "c94eb3e8-cfd0-46ca-8b24-6746f4437e4e" (UID: "c94eb3e8-cfd0-46ca-8b24-6746f4437e4e"). InnerVolumeSpecName "kube-api-access-k8qtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:40.852347 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.852308 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8qtp\" (UniqueName: \"kubernetes.io/projected/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-kube-api-access-k8qtp\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:40.852347 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.852341 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:40.852347 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.852352 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:40.852593 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:40.852360 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:41.084576 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:41.084539 5561 generic.go:356] "Generic (PLEG): container finished" podID="c94eb3e8-cfd0-46ca-8b24-6746f4437e4e" containerID="7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291" exitCode=0 Nov 02 20:42:41.084788 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:41.084588 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" Nov 02 20:42:41.084788 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:41.084595 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" event={"ID":"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e","Type":"ContainerDied","Data":"7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291"} Nov 02 20:42:41.084788 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:41.084646 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh" event={"ID":"c94eb3e8-cfd0-46ca-8b24-6746f4437e4e","Type":"ContainerDied","Data":"48cead0bac28b0fc272ff085e7603cfcf03876e6012e45c3a238d23792aa43d8"} Nov 02 20:42:41.084788 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:41.084677 5561 scope.go:122] "RemoveContainer" containerID="7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291" Nov 02 20:42:41.094155 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:41.094134 5561 scope.go:122] "RemoveContainer" containerID="7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291" Nov 02 20:42:41.094411 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:42:41.094394 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291\": container with ID starting with 7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291 not found: ID does not exist" containerID="7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291" Nov 02 20:42:41.094479 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:41.094419 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291"} err="failed to get container status \"7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291\": rpc error: code = NotFound desc = could not find container \"7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291\": container with ID starting with 7dfa32f8f5b247cb5e6204bc771be16ae1a906af9f0e78541f2adfdb5188a291 not found: ID does not exist" Nov 02 20:42:41.239744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:41.239695 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh"] Nov 02 20:42:41.242367 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:41.242340 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ae8e6e-5f4f76b956-r4llh"] Nov 02 20:42:41.312320 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:41.312285 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94eb3e8-cfd0-46ca-8b24-6746f4437e4e" path="/var/lib/kubelet/pods/c94eb3e8-cfd0-46ca-8b24-6746f4437e4e/volumes" Nov 02 20:42:42.044656 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.044622 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:42.059157 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.059120 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" podUID="65cfe3fa-bddf-4613-8987-9ebc09c064b4" containerName="main" containerID="cri-o://d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991" gracePeriod=45 Nov 02 20:42:42.224704 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.224673 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:42.306283 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.306223 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:42:42.367095 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.367069 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tmp-dir\") pod \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " Nov 02 20:42:42.367267 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.367107 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-dshm\") pod \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " Nov 02 20:42:42.367267 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.367133 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cfe3fa-bddf-4613-8987-9ebc09c064b4-kube-api-access-v9nv2\" (UniqueName: \"kubernetes.io/projected/65cfe3fa-bddf-4613-8987-9ebc09c064b4-kube-api-access-v9nv2\") pod \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " Nov 02 20:42:42.367267 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.367190 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-home\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-home\") pod \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " Nov 02 20:42:42.367267 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.367261 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tls-certs\") pod \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " Nov 02 20:42:42.367495 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.367318 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-model-cache\") pod \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\" (UID: \"65cfe3fa-bddf-4613-8987-9ebc09c064b4\") " Nov 02 20:42:42.367495 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.367376 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tmp-dir" pod "65cfe3fa-bddf-4613-8987-9ebc09c064b4" (UID: "65cfe3fa-bddf-4613-8987-9ebc09c064b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:42.367652 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.367629 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-model-cache" pod "65cfe3fa-bddf-4613-8987-9ebc09c064b4" (UID: "65cfe3fa-bddf-4613-8987-9ebc09c064b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:42.367853 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.367834 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:42.367967 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.367943 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-home" pod "65cfe3fa-bddf-4613-8987-9ebc09c064b4" (UID: "65cfe3fa-bddf-4613-8987-9ebc09c064b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:42.369517 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.369493 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-dshm" pod "65cfe3fa-bddf-4613-8987-9ebc09c064b4" (UID: "65cfe3fa-bddf-4613-8987-9ebc09c064b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:42.369625 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.369572 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tls-certs" pod "65cfe3fa-bddf-4613-8987-9ebc09c064b4" (UID: "65cfe3fa-bddf-4613-8987-9ebc09c064b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:42.369625 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.369584 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cfe3fa-bddf-4613-8987-9ebc09c064b4-kube-api-access-v9nv2" pod "65cfe3fa-bddf-4613-8987-9ebc09c064b4" (UID: "65cfe3fa-bddf-4613-8987-9ebc09c064b4"). InnerVolumeSpecName "kube-api-access-v9nv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:42.399474 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.399450 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:42.468367 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.468334 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:42.468367 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.468365 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9nv2\" (UniqueName: \"kubernetes.io/projected/65cfe3fa-bddf-4613-8987-9ebc09c064b4-kube-api-access-v9nv2\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:42.468566 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.468376 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:42.468566 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.468386 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65cfe3fa-bddf-4613-8987-9ebc09c064b4-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:42.468566 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:42.468395 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65cfe3fa-bddf-4613-8987-9ebc09c064b4-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:43.018165 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:43.018130 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:43.096431 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:43.096392 5561 generic.go:356] "Generic (PLEG): container finished" podID="65cfe3fa-bddf-4613-8987-9ebc09c064b4" containerID="d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991" exitCode=0 Nov 02 20:42:43.096854 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:43.096434 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" event={"ID":"65cfe3fa-bddf-4613-8987-9ebc09c064b4","Type":"ContainerDied","Data":"d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991"} Nov 02 20:42:43.096854 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:43.096442 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" Nov 02 20:42:43.096854 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:43.096474 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78" event={"ID":"65cfe3fa-bddf-4613-8987-9ebc09c064b4","Type":"ContainerDied","Data":"c6090eedba752c4c65fb5912e1fdb9ad3d891ceeadb18be21092b75dabf70213"} Nov 02 20:42:43.096854 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:43.096496 5561 scope.go:122] "RemoveContainer" containerID="d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991" Nov 02 20:42:43.108633 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:43.108614 5561 scope.go:122] "RemoveContainer" containerID="d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991" Nov 02 20:42:43.108989 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:42:43.108953 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991\": container with ID starting with d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991 not found: ID does not exist" containerID="d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991" Nov 02 20:42:43.109078 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:43.108997 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991"} err="failed to get container status \"d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991\": rpc error: code = NotFound desc = could not find container \"d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991\": container with ID starting with d84558522e161c8526afea91ade536fc668c9c3b9854510c2c5c776179bac991 not found: ID does not exist" Nov 02 20:42:43.251356 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:43.251314 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78"] Nov 02 20:42:43.298575 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:43.298493 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-aaa3e7/test-model-aaa3e7-kserve-69b4c96d67-ffk78"] Nov 02 20:42:43.312224 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:43.312187 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65cfe3fa-bddf-4613-8987-9ebc09c064b4" path="/var/lib/kubelet/pods/65cfe3fa-bddf-4613-8987-9ebc09c064b4/volumes" Nov 02 20:42:45.018475 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:45.018443 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:45.297460 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:45.297367 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:45.947347 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:45.947314 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:47.283092 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:47.283048 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:49.366361 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:49.366331 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:49.542760 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:49.542696 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:49.718614 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:49.718580 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:51.035949 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.035911 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:51.210591 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.210552 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:51.241451 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.241421 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52"] Nov 02 20:42:51.241693 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.241666 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" podUID="e8ab9f5e-0844-477c-ab31-58dfc7df6d7d" containerName="maas-api" containerID="cri-o://18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b" gracePeriod=30 Nov 02 20:42:51.384122 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.384065 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:51.497542 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.497516 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:42:51.552232 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.552194 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-maas-api-tls\") pod \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " Nov 02 20:42:51.552405 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.552283 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-metrics-certs\") pod \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " Nov 02 20:42:51.552405 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.552322 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-kube-api-access-z24x2\" (UniqueName: \"kubernetes.io/projected/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-kube-api-access-z24x2\") pod \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " Nov 02 20:42:51.552494 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.552422 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-openshift-service-ca\") pod \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\" (UID: \"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d\") " Nov 02 20:42:51.552824 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.552788 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-openshift-service-ca" pod "e8ab9f5e-0844-477c-ab31-58dfc7df6d7d" (UID: "e8ab9f5e-0844-477c-ab31-58dfc7df6d7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:51.554469 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.554444 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-metrics-certs" pod "e8ab9f5e-0844-477c-ab31-58dfc7df6d7d" (UID: "e8ab9f5e-0844-477c-ab31-58dfc7df6d7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:51.554819 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.554798 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-maas-api-tls" pod "e8ab9f5e-0844-477c-ab31-58dfc7df6d7d" (UID: "e8ab9f5e-0844-477c-ab31-58dfc7df6d7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:51.554893 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.554861 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-kube-api-access-z24x2" pod "e8ab9f5e-0844-477c-ab31-58dfc7df6d7d" (UID: "e8ab9f5e-0844-477c-ab31-58dfc7df6d7d"). InnerVolumeSpecName "kube-api-access-z24x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:51.653940 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.653913 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:51.653940 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.653940 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:51.654206 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.653950 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z24x2\" (UniqueName: \"kubernetes.io/projected/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-kube-api-access-z24x2\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:51.654206 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:51.653959 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:52.042286 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:52.042198 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:52.139916 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:52.139867 5561 generic.go:356] "Generic (PLEG): container finished" podID="e8ab9f5e-0844-477c-ab31-58dfc7df6d7d" containerID="18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b" exitCode=0 Nov 02 20:42:52.140086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:52.139927 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" Nov 02 20:42:52.140086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:52.139927 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" event={"ID":"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d","Type":"ContainerDied","Data":"18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b"} Nov 02 20:42:52.140086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:52.140043 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52" event={"ID":"e8ab9f5e-0844-477c-ab31-58dfc7df6d7d","Type":"ContainerDied","Data":"4fb7509478228728595c878a245b6fcccac9b0814e69e8bb19534a750f4b6a74"} Nov 02 20:42:52.140209 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:52.140066 5561 scope.go:122] "RemoveContainer" containerID="18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b" Nov 02 20:42:52.149283 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:52.149267 5561 scope.go:122] "RemoveContainer" containerID="18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b" Nov 02 20:42:52.149554 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:42:52.149531 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b\": container with ID starting with 18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b not found: ID does not exist" containerID="18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b" Nov 02 20:42:52.149619 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:52.149565 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b"} err="failed to get container status \"18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b\": rpc error: code = NotFound desc = could not find container \"18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b\": container with ID starting with 18962d48e2539681a26211cd78695a1ee64bd1a851d9b98ddb0039d438443d2b not found: ID does not exist" Nov 02 20:42:52.249894 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:52.249842 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:52.295026 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:52.294946 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52"] Nov 02 20:42:52.301974 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:52.301946 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bb78ba-f97d94f86-dht52"] Nov 02 20:42:52.422936 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:52.422899 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:53.116291 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.116257 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:53.312250 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.312218 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ab9f5e-0844-477c-ab31-58dfc7df6d7d" path="/var/lib/kubelet/pods/e8ab9f5e-0844-477c-ab31-58dfc7df6d7d/volumes" Nov 02 20:42:53.593707 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.593664 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" podUID="de249c08-26bd-4582-b4c6-dce05a1ae3b5" containerName="main" containerID="cri-o://5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe" gracePeriod=45 Nov 02 20:42:53.838501 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.838476 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:42:53.975755 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.975718 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-home\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-home\") pod \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " Nov 02 20:42:53.975960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.975805 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tls-certs\") pod \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " Nov 02 20:42:53.975960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.975834 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-dshm\") pod \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " Nov 02 20:42:53.975960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.975885 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-model-cache\") pod \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " Nov 02 20:42:53.975960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.975917 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tmp-dir\") pod \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " Nov 02 20:42:53.975960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.975954 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de249c08-26bd-4582-b4c6-dce05a1ae3b5-kube-api-access-x6dmr\" (UniqueName: \"kubernetes.io/projected/de249c08-26bd-4582-b4c6-dce05a1ae3b5-kube-api-access-x6dmr\") pod \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\" (UID: \"de249c08-26bd-4582-b4c6-dce05a1ae3b5\") " Nov 02 20:42:53.976200 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.976034 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-home" pod "de249c08-26bd-4582-b4c6-dce05a1ae3b5" (UID: "de249c08-26bd-4582-b4c6-dce05a1ae3b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:53.976200 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.976133 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-model-cache" pod "de249c08-26bd-4582-b4c6-dce05a1ae3b5" (UID: "de249c08-26bd-4582-b4c6-dce05a1ae3b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:53.976200 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.976165 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tmp-dir" pod "de249c08-26bd-4582-b4c6-dce05a1ae3b5" (UID: "de249c08-26bd-4582-b4c6-dce05a1ae3b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:53.976348 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.976296 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:53.976348 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.976316 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:53.976348 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.976328 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:53.978163 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.978137 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-dshm" pod "de249c08-26bd-4582-b4c6-dce05a1ae3b5" (UID: "de249c08-26bd-4582-b4c6-dce05a1ae3b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:53.978266 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.978190 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tls-certs" pod "de249c08-26bd-4582-b4c6-dce05a1ae3b5" (UID: "de249c08-26bd-4582-b4c6-dce05a1ae3b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:53.978266 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:53.978223 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de249c08-26bd-4582-b4c6-dce05a1ae3b5-kube-api-access-x6dmr" pod "de249c08-26bd-4582-b4c6-dce05a1ae3b5" (UID: "de249c08-26bd-4582-b4c6-dce05a1ae3b5"). InnerVolumeSpecName "kube-api-access-x6dmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:54.076986 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:54.076947 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de249c08-26bd-4582-b4c6-dce05a1ae3b5-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:54.076986 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:54.076977 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de249c08-26bd-4582-b4c6-dce05a1ae3b5-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:54.076986 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:54.076987 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6dmr\" (UniqueName: \"kubernetes.io/projected/de249c08-26bd-4582-b4c6-dce05a1ae3b5-kube-api-access-x6dmr\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:54.152344 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:54.152306 5561 generic.go:356] "Generic (PLEG): container finished" podID="de249c08-26bd-4582-b4c6-dce05a1ae3b5" containerID="5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe" exitCode=0 Nov 02 20:42:54.152815 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:54.152361 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" Nov 02 20:42:54.152815 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:54.152390 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" event={"ID":"de249c08-26bd-4582-b4c6-dce05a1ae3b5","Type":"ContainerDied","Data":"5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe"} Nov 02 20:42:54.152815 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:54.152435 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p" event={"ID":"de249c08-26bd-4582-b4c6-dce05a1ae3b5","Type":"ContainerDied","Data":"062512e06016491edaeee3521b6274f21d5ede233d96a9c15e40c801e287955d"} Nov 02 20:42:54.152815 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:54.152454 5561 scope.go:122] "RemoveContainer" containerID="5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe" Nov 02 20:42:54.165796 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:54.165768 5561 scope.go:122] "RemoveContainer" containerID="5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe" Nov 02 20:42:54.166098 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:42:54.166079 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe\": container with ID starting with 5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe not found: ID does not exist" containerID="5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe" Nov 02 20:42:54.166185 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:54.166106 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe"} err="failed to get container status \"5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe\": rpc error: code = NotFound desc = could not find container \"5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe\": container with ID starting with 5d53afd2d9774936a1716beec4d22675ee9d99788f90bbcab8aad186001a8fbe not found: ID does not exist" Nov 02 20:42:54.308026 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:54.307937 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p"] Nov 02 20:42:54.311321 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:54.311297 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2b3b9e/test-model-2b3b9e-kserve-6f654f955c-25r8p"] Nov 02 20:42:55.203230 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:55.203194 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:55.312072 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:55.312039 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de249c08-26bd-4582-b4c6-dce05a1ae3b5" path="/var/lib/kubelet/pods/de249c08-26bd-4582-b4c6-dce05a1ae3b5/volumes" Nov 02 20:42:55.545400 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:55.545316 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:56.065418 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:56.065387 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:58.003479 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:58.003444 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:58.199161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:58.199124 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:58.510443 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:42:58.510412 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:01.857561 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:01.857528 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:02.029473 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:02.029437 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:02.202644 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:02.202612 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:02.767888 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:02.767849 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:03.778663 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:43:03.778628 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:43:04.448141 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:04.448104 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:04.620394 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:04.620358 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:06.751443 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:06.751410 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:07.794724 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:07.794689 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:11.919089 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:11.919051 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:13.497983 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:13.497953 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:13.749720 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:13.749633 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:13.925371 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:13.925335 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:14.389759 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:14.389726 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:15.250352 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:15.250317 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn"] Nov 02 20:43:15.361776 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:15.361736 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8"] Nov 02 20:43:15.664590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:15.664558 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:15.832746 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:15.832712 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:18.553268 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:18.553233 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:20.251622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:20.251580 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" podUID="cf0ee610-89f1-4e6f-9017-1d3777be0b29" containerName="payload-processing" containerID="cri-o://8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4" gracePeriod=25 Nov 02 20:43:20.255352 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:20.255319 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:20.362818 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:20.362773 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" podUID="39722652-3460-46ed-89ce-5ab8a3eb255d" containerName="payload-pre-processing" containerID="cri-o://d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40" gracePeriod=25 Nov 02 20:43:20.430211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:20.430173 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:20.607373 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:20.607281 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:21.050208 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:21.050160 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:23.076409 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:23.076373 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:23.386339 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:23.386248 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:24.012342 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:24.012299 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" podUID="39722652-3460-46ed-89ce-5ab8a3eb255d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:43:25.017693 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:25.017652 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" podUID="cf0ee610-89f1-4e6f-9017-1d3777be0b29" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:43:25.968106 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:25.968055 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x"] Nov 02 20:43:26.014868 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:26.014829 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw"] Nov 02 20:43:26.090044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:26.090005 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:28.199767 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:28.199733 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:32.000668 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.000629 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv"] Nov 02 20:43:32.001115 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.000843 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" podUID="0db8b320-9073-44fd-b41e-cb96b8e76fcf" containerName="maas-api" containerID="cri-o://b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae" gracePeriod=30 Nov 02 20:43:32.142231 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.142189 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:32.270595 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.270515 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:43:32.312621 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.312575 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-maas-api-tls\") pod \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " Nov 02 20:43:32.312798 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.312699 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-metrics-certs\") pod \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " Nov 02 20:43:32.312798 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.312740 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db8b320-9073-44fd-b41e-cb96b8e76fcf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db8b320-9073-44fd-b41e-cb96b8e76fcf-openshift-service-ca\") pod \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " Nov 02 20:43:32.312798 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.312780 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db8b320-9073-44fd-b41e-cb96b8e76fcf-kube-api-access-dq68q\" (UniqueName: \"kubernetes.io/projected/0db8b320-9073-44fd-b41e-cb96b8e76fcf-kube-api-access-dq68q\") pod \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\" (UID: \"0db8b320-9073-44fd-b41e-cb96b8e76fcf\") " Nov 02 20:43:32.313209 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.313177 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db8b320-9073-44fd-b41e-cb96b8e76fcf-openshift-service-ca" pod "0db8b320-9073-44fd-b41e-cb96b8e76fcf" (UID: "0db8b320-9073-44fd-b41e-cb96b8e76fcf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:32.314694 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.314669 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-maas-api-tls" pod "0db8b320-9073-44fd-b41e-cb96b8e76fcf" (UID: "0db8b320-9073-44fd-b41e-cb96b8e76fcf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:32.314895 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.314853 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db8b320-9073-44fd-b41e-cb96b8e76fcf-kube-api-access-dq68q" pod "0db8b320-9073-44fd-b41e-cb96b8e76fcf" (UID: "0db8b320-9073-44fd-b41e-cb96b8e76fcf"). InnerVolumeSpecName "kube-api-access-dq68q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:32.314980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.314933 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-metrics-certs" pod "0db8b320-9073-44fd-b41e-cb96b8e76fcf" (UID: "0db8b320-9073-44fd-b41e-cb96b8e76fcf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:32.326129 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.326096 5561 generic.go:356] "Generic (PLEG): container finished" podID="0db8b320-9073-44fd-b41e-cb96b8e76fcf" containerID="b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae" exitCode=0 Nov 02 20:43:32.326251 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.326142 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" Nov 02 20:43:32.326251 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.326146 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" event={"ID":"0db8b320-9073-44fd-b41e-cb96b8e76fcf","Type":"ContainerDied","Data":"b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae"} Nov 02 20:43:32.326251 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.326209 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv" event={"ID":"0db8b320-9073-44fd-b41e-cb96b8e76fcf","Type":"ContainerDied","Data":"4324772e00cb8a29cab39034c1ac273acb457d40bc8a9d627bbe7f17cc4557bf"} Nov 02 20:43:32.326251 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.326238 5561 scope.go:122] "RemoveContainer" containerID="b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae" Nov 02 20:43:32.335128 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.335107 5561 scope.go:122] "RemoveContainer" containerID="b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae" Nov 02 20:43:32.335426 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:43:32.335405 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae\": container with ID starting with b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae not found: ID does not exist" containerID="b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae" Nov 02 20:43:32.335499 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.335434 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae"} err="failed to get container status \"b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae\": rpc error: code = NotFound desc = could not find container \"b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae\": container with ID starting with b5cb1befdc010bc8955c74ab213474b40dd62817d08348c911eb2a7e317dcdae not found: ID does not exist" Nov 02 20:43:32.414037 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.414007 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:32.414037 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.414035 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0db8b320-9073-44fd-b41e-cb96b8e76fcf-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:32.414037 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.414045 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq68q\" (UniqueName: \"kubernetes.io/projected/0db8b320-9073-44fd-b41e-cb96b8e76fcf-kube-api-access-dq68q\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:32.414268 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.414054 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0db8b320-9073-44fd-b41e-cb96b8e76fcf-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:32.483993 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.483955 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv"] Nov 02 20:43:32.485298 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:32.485280 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2b3b9e-69dfc9d6d4-b5dsv"] Nov 02 20:43:33.312661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:33.312623 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db8b320-9073-44fd-b41e-cb96b8e76fcf" path="/var/lib/kubelet/pods/0db8b320-9073-44fd-b41e-cb96b8e76fcf/volumes" Nov 02 20:43:33.404745 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:33.404709 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:33.583607 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:33.583505 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:33.778704 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:43:33.778662 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:43:33.833441 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:33.833409 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:34.012199 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:34.012160 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" podUID="39722652-3460-46ed-89ce-5ab8a3eb255d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:43:34.033320 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:34.033284 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:34.460248 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:34.460217 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:35.017814 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:35.017772 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" podUID="cf0ee610-89f1-4e6f-9017-1d3777be0b29" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:43:38.867619 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:38.867583 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:40.245056 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.245026 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:40.427585 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.424829 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:40.519749 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.519709 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" Nov 02 20:43:40.592488 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.592446 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf0ee610-89f1-4e6f-9017-1d3777be0b29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0ee610-89f1-4e6f-9017-1d3777be0b29-plugins-config-volume\") pod \"cf0ee610-89f1-4e6f-9017-1d3777be0b29\" (UID: \"cf0ee610-89f1-4e6f-9017-1d3777be0b29\") " Nov 02 20:43:40.592665 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.592537 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0ee610-89f1-4e6f-9017-1d3777be0b29-kube-api-access-dfmj9\" (UniqueName: \"kubernetes.io/projected/cf0ee610-89f1-4e6f-9017-1d3777be0b29-kube-api-access-dfmj9\") pod \"cf0ee610-89f1-4e6f-9017-1d3777be0b29\" (UID: \"cf0ee610-89f1-4e6f-9017-1d3777be0b29\") " Nov 02 20:43:40.592816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.592792 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0ee610-89f1-4e6f-9017-1d3777be0b29-plugins-config-volume" pod "cf0ee610-89f1-4e6f-9017-1d3777be0b29" (UID: "cf0ee610-89f1-4e6f-9017-1d3777be0b29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:40.594665 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.594640 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0ee610-89f1-4e6f-9017-1d3777be0b29-kube-api-access-dfmj9" pod "cf0ee610-89f1-4e6f-9017-1d3777be0b29" (UID: "cf0ee610-89f1-4e6f-9017-1d3777be0b29"). InnerVolumeSpecName "kube-api-access-dfmj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:40.595896 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.595857 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:40.608693 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.608670 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" Nov 02 20:43:40.693924 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.693871 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39722652-3460-46ed-89ce-5ab8a3eb255d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39722652-3460-46ed-89ce-5ab8a3eb255d-plugins-config-volume\") pod \"39722652-3460-46ed-89ce-5ab8a3eb255d\" (UID: \"39722652-3460-46ed-89ce-5ab8a3eb255d\") " Nov 02 20:43:40.694123 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.694037 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39722652-3460-46ed-89ce-5ab8a3eb255d-kube-api-access-p45h6\" (UniqueName: \"kubernetes.io/projected/39722652-3460-46ed-89ce-5ab8a3eb255d-kube-api-access-p45h6\") pod \"39722652-3460-46ed-89ce-5ab8a3eb255d\" (UID: \"39722652-3460-46ed-89ce-5ab8a3eb255d\") " Nov 02 20:43:40.694224 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.694210 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfmj9\" (UniqueName: \"kubernetes.io/projected/cf0ee610-89f1-4e6f-9017-1d3777be0b29-kube-api-access-dfmj9\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:40.694284 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.694226 5561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0ee610-89f1-4e6f-9017-1d3777be0b29-plugins-config-volume\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:40.694284 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.694246 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39722652-3460-46ed-89ce-5ab8a3eb255d-plugins-config-volume" pod "39722652-3460-46ed-89ce-5ab8a3eb255d" (UID: "39722652-3460-46ed-89ce-5ab8a3eb255d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:40.696105 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.696086 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39722652-3460-46ed-89ce-5ab8a3eb255d-kube-api-access-p45h6" pod "39722652-3460-46ed-89ce-5ab8a3eb255d" (UID: "39722652-3460-46ed-89ce-5ab8a3eb255d"). InnerVolumeSpecName "kube-api-access-p45h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:40.795536 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.795451 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p45h6\" (UniqueName: \"kubernetes.io/projected/39722652-3460-46ed-89ce-5ab8a3eb255d-kube-api-access-p45h6\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:40.795536 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.795482 5561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39722652-3460-46ed-89ce-5ab8a3eb255d-plugins-config-volume\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:40.988509 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:40.988305 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" containerID="cri-o://b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b" gracePeriod=45 Nov 02 20:43:41.034489 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.034444 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" containerID="cri-o://32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172" gracePeriod=45 Nov 02 20:43:41.092145 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.092064 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:41.222001 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.221962 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": dial tcp 10.134.0.41:8000: connect: connection refused" Nov 02 20:43:41.265654 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.265626 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:43:41.307211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.307187 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:43:41.369773 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.369677 5561 generic.go:356] "Generic (PLEG): container finished" podID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerID="b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b" exitCode=0 Nov 02 20:43:41.369944 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.369783 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" Nov 02 20:43:41.369944 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.369805 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" event={"ID":"0475c8c4-b045-4faf-b21f-6501ce60a3df","Type":"ContainerDied","Data":"b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b"} Nov 02 20:43:41.369944 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.369855 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x" event={"ID":"0475c8c4-b045-4faf-b21f-6501ce60a3df","Type":"ContainerDied","Data":"6df4c28c0646421cfdbc0f8bf9b228dc1d35cf686d44fcc0d11de95399239e5f"} Nov 02 20:43:41.369944 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.369894 5561 scope.go:122] "RemoveContainer" containerID="b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b" Nov 02 20:43:41.371188 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.371151 5561 generic.go:356] "Generic (PLEG): container finished" podID="cf0ee610-89f1-4e6f-9017-1d3777be0b29" containerID="8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4" exitCode=0 Nov 02 20:43:41.371249 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.371196 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" Nov 02 20:43:41.371249 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.371234 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" event={"ID":"cf0ee610-89f1-4e6f-9017-1d3777be0b29","Type":"ContainerDied","Data":"8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4"} Nov 02 20:43:41.371354 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.371294 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn" event={"ID":"cf0ee610-89f1-4e6f-9017-1d3777be0b29","Type":"ContainerDied","Data":"73ed0dd9c968bd886070c4c0f6be46d809414ad668f51fe024709520a7dfc0ef"} Nov 02 20:43:41.372694 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.372671 5561 generic.go:356] "Generic (PLEG): container finished" podID="39722652-3460-46ed-89ce-5ab8a3eb255d" containerID="d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40" exitCode=0 Nov 02 20:43:41.372816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.372715 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" Nov 02 20:43:41.372816 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.372754 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" event={"ID":"39722652-3460-46ed-89ce-5ab8a3eb255d","Type":"ContainerDied","Data":"d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40"} Nov 02 20:43:41.373399 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.373372 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8" event={"ID":"39722652-3460-46ed-89ce-5ab8a3eb255d","Type":"ContainerDied","Data":"6c8346316cf317ab55d29625ad242d638f89eb9b6c20fd6bb23c492cc1f0d0d7"} Nov 02 20:43:41.375009 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.374984 5561 generic.go:356] "Generic (PLEG): container finished" podID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerID="32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172" exitCode=0 Nov 02 20:43:41.375095 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.375020 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" event={"ID":"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa","Type":"ContainerDied","Data":"32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172"} Nov 02 20:43:41.375095 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.375045 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" event={"ID":"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa","Type":"ContainerDied","Data":"f916385e43ae57e830d6134b155da557c33afb1daeb1a2b8ae1997ab531f9f48"} Nov 02 20:43:41.375197 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.375101 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw" Nov 02 20:43:41.382898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.382834 5561 scope.go:122] "RemoveContainer" containerID="b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b" Nov 02 20:43:41.383544 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:43:41.383509 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b\": container with ID starting with b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b not found: ID does not exist" containerID="b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b" Nov 02 20:43:41.383653 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.383557 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b"} err="failed to get container status \"b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b\": rpc error: code = NotFound desc = could not find container \"b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b\": container with ID starting with b56c238332ce80653aca3773fe58d835d9401f9411697de12a0a82de321a2c2b not found: ID does not exist" Nov 02 20:43:41.383653 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.383583 5561 scope.go:122] "RemoveContainer" containerID="8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4" Nov 02 20:43:41.394801 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.394780 5561 scope.go:122] "RemoveContainer" containerID="8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4" Nov 02 20:43:41.395265 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:43:41.395236 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4\": container with ID starting with 8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4 not found: ID does not exist" containerID="8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4" Nov 02 20:43:41.395376 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.395275 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4"} err="failed to get container status \"8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4\": rpc error: code = NotFound desc = could not find container \"8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4\": container with ID starting with 8a7edcece30552bbfaa2b411bf1e92aa742277641c17d8b1071e634866febee4 not found: ID does not exist" Nov 02 20:43:41.395376 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.395302 5561 scope.go:122] "RemoveContainer" containerID="d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40" Nov 02 20:43:41.401039 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.401015 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-dshm\") pod \"0475c8c4-b045-4faf-b21f-6501ce60a3df\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " Nov 02 20:43:41.401152 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.401054 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0475c8c4-b045-4faf-b21f-6501ce60a3df-tls-certs\" (UniqueName: \"kubernetes.io/secret/0475c8c4-b045-4faf-b21f-6501ce60a3df-tls-certs\") pod \"0475c8c4-b045-4faf-b21f-6501ce60a3df\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " Nov 02 20:43:41.401152 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.401082 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-model-cache\") pod \"0475c8c4-b045-4faf-b21f-6501ce60a3df\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " Nov 02 20:43:41.401152 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.401118 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0475c8c4-b045-4faf-b21f-6501ce60a3df-kube-api-access-ds56t\" (UniqueName: \"kubernetes.io/projected/0475c8c4-b045-4faf-b21f-6501ce60a3df-kube-api-access-ds56t\") pod \"0475c8c4-b045-4faf-b21f-6501ce60a3df\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " Nov 02 20:43:41.401332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.401164 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-home\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-home\") pod \"0475c8c4-b045-4faf-b21f-6501ce60a3df\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " Nov 02 20:43:41.401332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.401205 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-dshm\") pod \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " Nov 02 20:43:41.401332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.401259 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tls-certs\") pod \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " Nov 02 20:43:41.401332 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.401288 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-kube-api-access-rvt84\" (UniqueName: \"kubernetes.io/projected/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-kube-api-access-rvt84\") pod \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " Nov 02 20:43:41.401548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.401345 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-tmp-dir\") pod \"0475c8c4-b045-4faf-b21f-6501ce60a3df\" (UID: \"0475c8c4-b045-4faf-b21f-6501ce60a3df\") " Nov 02 20:43:41.401548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.401391 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tmp-dir\") pod \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " Nov 02 20:43:41.401548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.401459 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-model-cache\") pod \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " Nov 02 20:43:41.401548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.401488 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-home\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-home\") pod \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\" (UID: \"46dbd4e4-a31e-4587-90f2-8bb3ad0832aa\") " Nov 02 20:43:41.402229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.402127 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tmp-dir" pod "46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" (UID: "46dbd4e4-a31e-4587-90f2-8bb3ad0832aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:41.402229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.402149 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-home" pod "0475c8c4-b045-4faf-b21f-6501ce60a3df" (UID: "0475c8c4-b045-4faf-b21f-6501ce60a3df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:41.402468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.402369 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-model-cache" pod "46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" (UID: "46dbd4e4-a31e-4587-90f2-8bb3ad0832aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:41.402468 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.402375 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-tmp-dir" pod "0475c8c4-b045-4faf-b21f-6501ce60a3df" (UID: "0475c8c4-b045-4faf-b21f-6501ce60a3df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:41.402634 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.402485 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-home" pod "46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" (UID: "46dbd4e4-a31e-4587-90f2-8bb3ad0832aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:41.402634 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.402612 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-model-cache" pod "0475c8c4-b045-4faf-b21f-6501ce60a3df" (UID: "0475c8c4-b045-4faf-b21f-6501ce60a3df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:41.404588 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.404559 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tls-certs" pod "46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" (UID: "46dbd4e4-a31e-4587-90f2-8bb3ad0832aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:41.404693 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.404618 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0475c8c4-b045-4faf-b21f-6501ce60a3df-tls-certs" pod "0475c8c4-b045-4faf-b21f-6501ce60a3df" (UID: "0475c8c4-b045-4faf-b21f-6501ce60a3df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:41.404773 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.404754 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-dshm" pod "0475c8c4-b045-4faf-b21f-6501ce60a3df" (UID: "0475c8c4-b045-4faf-b21f-6501ce60a3df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:41.404832 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.404814 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-dshm" pod "46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" (UID: "46dbd4e4-a31e-4587-90f2-8bb3ad0832aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:41.411955 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.405100 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-kube-api-access-rvt84" pod "46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" (UID: "46dbd4e4-a31e-4587-90f2-8bb3ad0832aa"). InnerVolumeSpecName "kube-api-access-rvt84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:41.411955 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.406070 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0475c8c4-b045-4faf-b21f-6501ce60a3df-kube-api-access-ds56t" pod "0475c8c4-b045-4faf-b21f-6501ce60a3df" (UID: "0475c8c4-b045-4faf-b21f-6501ce60a3df"). InnerVolumeSpecName "kube-api-access-ds56t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:41.411955 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.406153 5561 scope.go:122] "RemoveContainer" containerID="d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40" Nov 02 20:43:41.411955 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:43:41.406502 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40\": container with ID starting with d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40 not found: ID does not exist" containerID="d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40" Nov 02 20:43:41.411955 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.406529 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40"} err="failed to get container status \"d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40\": rpc error: code = NotFound desc = could not find container \"d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40\": container with ID starting with d23c4e8902d37ee0a58541b3b65161e821f2b648e7671765f0ca99b3f5870f40 not found: ID does not exist" Nov 02 20:43:41.411955 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.406548 5561 scope.go:122] "RemoveContainer" containerID="32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172" Nov 02 20:43:41.423843 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.423813 5561 scope.go:122] "RemoveContainer" containerID="32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172" Nov 02 20:43:41.424182 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:43:41.424162 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172\": container with ID starting with 32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172 not found: ID does not exist" containerID="32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172" Nov 02 20:43:41.424269 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.424190 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172"} err="failed to get container status \"32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172\": rpc error: code = NotFound desc = could not find container \"32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172\": container with ID starting with 32b1ded17d712d9050bc91b427f24cc7dc7c3b84d674a6aa4fe21c076a279172 not found: ID does not exist" Nov 02 20:43:41.502549 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.502511 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.502549 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.502543 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.502549 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.502552 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.502791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.502562 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.502791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.502572 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.502791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.502584 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0475c8c4-b045-4faf-b21f-6501ce60a3df-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.502791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.502596 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.502791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.502607 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds56t\" (UniqueName: \"kubernetes.io/projected/0475c8c4-b045-4faf-b21f-6501ce60a3df-kube-api-access-ds56t\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.502791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.502616 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0475c8c4-b045-4faf-b21f-6501ce60a3df-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.502791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.502623 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.502791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.502630 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.502791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.502638 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvt84\" (UniqueName: \"kubernetes.io/projected/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa-kube-api-access-rvt84\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.735774 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.735740 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8"] Nov 02 20:43:41.739199 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.739172 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b3b9e-5c8fdb77dc-qckf8"] Nov 02 20:43:41.892365 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.892320 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn"] Nov 02 20:43:41.893624 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:41.893599 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b3b9e-67f6fd5df7-nkvsn"] Nov 02 20:43:42.001598 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:42.001517 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw"] Nov 02 20:43:42.004721 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:42.004696 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/simulator-w4-cc64d3-kserve-555d8c859b-hbthw"] Nov 02 20:43:42.108135 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:42.107929 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x"] Nov 02 20:43:42.109977 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:42.109946 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/premium-w4-cc64d3-kserve-599c7c4c7-lrr9x"] Nov 02 20:43:43.311751 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:43.311712 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" path="/var/lib/kubelet/pods/0475c8c4-b045-4faf-b21f-6501ce60a3df/volumes" Nov 02 20:43:43.312272 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:43.312065 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39722652-3460-46ed-89ce-5ab8a3eb255d" path="/var/lib/kubelet/pods/39722652-3460-46ed-89ce-5ab8a3eb255d/volumes" Nov 02 20:43:43.312450 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:43.312433 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" path="/var/lib/kubelet/pods/46dbd4e4-a31e-4587-90f2-8bb3ad0832aa/volumes" Nov 02 20:43:43.313097 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:43.313081 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf0ee610-89f1-4e6f-9017-1d3777be0b29" path="/var/lib/kubelet/pods/cf0ee610-89f1-4e6f-9017-1d3777be0b29/volumes" Nov 02 20:43:43.550641 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:43.550606 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:46.123712 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:46.123676 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:48.469235 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:48.469201 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:53.050634 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:53.050598 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:53.415591 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:53.415559 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:53.605423 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:53.605391 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:57.073250 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.073215 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh"] Nov 02 20:43:57.073941 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.073406 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" podUID="1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d" containerName="maas-api" containerID="cri-o://3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351" gracePeriod=30 Nov 02 20:43:57.094813 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:43:57.094777 5561 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f198ce3_2bfb_4b2d_bda5_cd81ff90e37d.slice/crio-3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:43:57.107521 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.107489 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" podUID="1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.34:8443/health\": dial tcp 10.134.0.34:8443: connect: connection refused" Nov 02 20:43:57.216780 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.216744 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:57.333946 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.333864 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:43:57.445654 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.445622 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-kube-api-access-xx7qd\" (UniqueName: \"kubernetes.io/projected/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-kube-api-access-xx7qd\") pod \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " Nov 02 20:43:57.445866 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.445668 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-maas-api-tls\") pod \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " Nov 02 20:43:57.446092 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.446065 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-openshift-service-ca\") pod \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " Nov 02 20:43:57.446228 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.446114 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-metrics-certs\") pod \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\" (UID: \"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d\") " Nov 02 20:43:57.446504 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.446461 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-openshift-service-ca" pod "1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d" (UID: "1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:57.449483 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.449455 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-kube-api-access-xx7qd" pod "1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d" (UID: "1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d"). InnerVolumeSpecName "kube-api-access-xx7qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:57.449739 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.449717 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-maas-api-tls" pod "1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d" (UID: "1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:57.450559 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.450534 5561 generic.go:356] "Generic (PLEG): container finished" podID="1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d" containerID="3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351" exitCode=0 Nov 02 20:43:57.450666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.450583 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" Nov 02 20:43:57.450666 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.450580 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" event={"ID":"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d","Type":"ContainerDied","Data":"3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351"} Nov 02 20:43:57.450796 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.450699 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh" event={"ID":"1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d","Type":"ContainerDied","Data":"fc95320918028c8af32cba6ca8091e97455af7c39dcfccaf551b18b12b13eb5c"} Nov 02 20:43:57.450796 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.450731 5561 scope.go:122] "RemoveContainer" containerID="3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351" Nov 02 20:43:57.453291 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.453266 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-metrics-certs" pod "1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d" (UID: "1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:57.465684 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.465658 5561 scope.go:122] "RemoveContainer" containerID="3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351" Nov 02 20:43:57.466028 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:43:57.466002 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351\": container with ID starting with 3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351 not found: ID does not exist" containerID="3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351" Nov 02 20:43:57.466111 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.466035 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351"} err="failed to get container status \"3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351\": rpc error: code = NotFound desc = could not find container \"3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351\": container with ID starting with 3d2197a1e6c81ec1b3d67b3e7d509d21bcbac9e975776f6896834380d0b4b351 not found: ID does not exist" Nov 02 20:43:57.547540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.547509 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx7qd\" (UniqueName: \"kubernetes.io/projected/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-kube-api-access-xx7qd\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:57.547540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.547536 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:57.547540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.547545 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:57.547781 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.547553 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:57.864777 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.864740 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh"] Nov 02 20:43:57.912369 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:57.912334 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc64d3-7c954cd6cb-chhbh"] Nov 02 20:43:58.351644 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:58.351604 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:58.673971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:58.673936 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:58.926771 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:58.926670 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:59.311801 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:59.311727 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d" path="/var/lib/kubelet/pods/1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d/volumes" Nov 02 20:43:59.548538 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:43:59.548506 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:02.892189 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:02.892156 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:03.076905 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:03.076859 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:03.779418 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:44:03.779377 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:44:03.865655 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:03.865619 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:05.624057 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:05.624020 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:05.817034 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:05.816989 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:05.999064 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:05.999027 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:08.159730 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.159682 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7"] Nov 02 20:44:08.160302 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.159907 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" podUID="05959640-ae04-4597-b190-232c13f303c2" containerName="maas-api" containerID="cri-o://ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819" gracePeriod=30 Nov 02 20:44:08.307823 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.307710 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:08.409643 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.409615 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:44:08.501924 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.501820 5561 generic.go:356] "Generic (PLEG): container finished" podID="05959640-ae04-4597-b190-232c13f303c2" containerID="ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819" exitCode=0 Nov 02 20:44:08.501924 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.501870 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" Nov 02 20:44:08.501924 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.501909 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" event={"ID":"05959640-ae04-4597-b190-232c13f303c2","Type":"ContainerDied","Data":"ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819"} Nov 02 20:44:08.502217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.501971 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7" event={"ID":"05959640-ae04-4597-b190-232c13f303c2","Type":"ContainerDied","Data":"1d8d1ef69fc5f70f9d2decd5c087ac1a944f489d884f34ddefc9d2c941d560ab"} Nov 02 20:44:08.502217 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.502001 5561 scope.go:122] "RemoveContainer" containerID="ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819" Nov 02 20:44:08.512734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.512711 5561 scope.go:122] "RemoveContainer" containerID="ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819" Nov 02 20:44:08.513098 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:44:08.513064 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819\": container with ID starting with ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819 not found: ID does not exist" containerID="ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819" Nov 02 20:44:08.513219 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.513111 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819"} err="failed to get container status \"ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819\": rpc error: code = NotFound desc = could not find container \"ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819\": container with ID starting with ee521d4b9d2c62594ebb77db97eac5009403dc004a2cd0e877244cc414465819 not found: ID does not exist" Nov 02 20:44:08.549865 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.549828 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-maas-api-tls\") pod \"05959640-ae04-4597-b190-232c13f303c2\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " Nov 02 20:44:08.550050 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.549947 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05959640-ae04-4597-b190-232c13f303c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05959640-ae04-4597-b190-232c13f303c2-openshift-service-ca\") pod \"05959640-ae04-4597-b190-232c13f303c2\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " Nov 02 20:44:08.550050 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.549994 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-metrics-certs\") pod \"05959640-ae04-4597-b190-232c13f303c2\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " Nov 02 20:44:08.550050 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.550041 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05959640-ae04-4597-b190-232c13f303c2-kube-api-access-7g6fc\" (UniqueName: \"kubernetes.io/projected/05959640-ae04-4597-b190-232c13f303c2-kube-api-access-7g6fc\") pod \"05959640-ae04-4597-b190-232c13f303c2\" (UID: \"05959640-ae04-4597-b190-232c13f303c2\") " Nov 02 20:44:08.550434 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.550406 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05959640-ae04-4597-b190-232c13f303c2-openshift-service-ca" pod "05959640-ae04-4597-b190-232c13f303c2" (UID: "05959640-ae04-4597-b190-232c13f303c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:08.552205 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.552180 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-maas-api-tls" pod "05959640-ae04-4597-b190-232c13f303c2" (UID: "05959640-ae04-4597-b190-232c13f303c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:08.552311 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.552227 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05959640-ae04-4597-b190-232c13f303c2-kube-api-access-7g6fc" pod "05959640-ae04-4597-b190-232c13f303c2" (UID: "05959640-ae04-4597-b190-232c13f303c2"). InnerVolumeSpecName "kube-api-access-7g6fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:08.552394 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.552378 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-metrics-certs" pod "05959640-ae04-4597-b190-232c13f303c2" (UID: "05959640-ae04-4597-b190-232c13f303c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:08.650663 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.650629 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05959640-ae04-4597-b190-232c13f303c2-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:08.650663 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.650661 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:08.650663 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.650672 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g6fc\" (UniqueName: \"kubernetes.io/projected/05959640-ae04-4597-b190-232c13f303c2-kube-api-access-7g6fc\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:08.650927 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.650683 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05959640-ae04-4597-b190-232c13f303c2-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:08.920237 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.920198 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7"] Nov 02 20:44:08.921541 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:08.921498 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-035e47-cf755d9bc-kp8v7"] Nov 02 20:44:09.311401 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:09.311368 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05959640-ae04-4597-b190-232c13f303c2" path="/var/lib/kubelet/pods/05959640-ae04-4597-b190-232c13f303c2/volumes" Nov 02 20:44:09.625813 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:09.625736 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:09.805925 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:09.805868 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:09.997080 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:09.997051 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:10.593168 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:10.593136 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:16.377093 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:16.377060 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:16.558006 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:16.557966 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:16.735165 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:16.735127 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:22.942133 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:22.942097 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:23.120306 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:23.120271 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:23.293712 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:23.293623 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:28.944248 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:28.944194 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:33.779567 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:44:33.779526 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:44:59.561717 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.561685 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:59.705887 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.705832 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th"] Nov 02 20:44:59.706301 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706276 5561 cpu_manager.go:414] "containerMap: removing container" podUID="05959640-ae04-4597-b190-232c13f303c2" containerName="maas-api" Nov 02 20:44:59.706301 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706297 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05959640-ae04-4597-b190-232c13f303c2" containerName="maas-api" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706320 5561 cpu_manager.go:414] "containerMap: removing container" podUID="39722652-3460-46ed-89ce-5ab8a3eb255d" containerName="payload-pre-processing" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706328 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39722652-3460-46ed-89ce-5ab8a3eb255d" containerName="payload-pre-processing" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706342 5561 cpu_manager.go:414] "containerMap: removing container" podUID="65cfe3fa-bddf-4613-8987-9ebc09c064b4" containerName="main" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706351 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cfe3fa-bddf-4613-8987-9ebc09c064b4" containerName="main" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706368 5561 cpu_manager.go:414] "containerMap: removing container" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706375 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706386 5561 cpu_manager.go:414] "containerMap: removing container" podUID="b90f9ab7-7575-4bd8-b55b-93872c815747" containerName="payload-pre-processing" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706393 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90f9ab7-7575-4bd8-b55b-93872c815747" containerName="payload-pre-processing" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706404 5561 cpu_manager.go:414] "containerMap: removing container" podUID="20cd41d9-5f2d-4ece-a7f7-e62487149e0a" containerName="main" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706411 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20cd41d9-5f2d-4ece-a7f7-e62487149e0a" containerName="main" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706426 5561 cpu_manager.go:414] "containerMap: removing container" podUID="cf0ee610-89f1-4e6f-9017-1d3777be0b29" containerName="payload-processing" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706434 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0ee610-89f1-4e6f-9017-1d3777be0b29" containerName="payload-processing" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706446 5561 cpu_manager.go:414] "containerMap: removing container" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706453 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706475 5561 cpu_manager.go:414] "containerMap: removing container" podUID="c94eb3e8-cfd0-46ca-8b24-6746f4437e4e" containerName="maas-api" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706482 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94eb3e8-cfd0-46ca-8b24-6746f4437e4e" containerName="maas-api" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706492 5561 cpu_manager.go:414] "containerMap: removing container" podUID="fa33f7d5-310c-4581-84c4-317226e57817" containerName="istio-proxy" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706499 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa33f7d5-310c-4581-84c4-317226e57817" containerName="istio-proxy" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706511 5561 cpu_manager.go:414] "containerMap: removing container" podUID="de249c08-26bd-4582-b4c6-dce05a1ae3b5" containerName="main" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706519 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de249c08-26bd-4582-b4c6-dce05a1ae3b5" containerName="main" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706528 5561 cpu_manager.go:414] "containerMap: removing container" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" Nov 02 20:44:59.706528 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706537 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706547 5561 cpu_manager.go:414] "containerMap: removing container" podUID="1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d" containerName="maas-api" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706554 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d" containerName="maas-api" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706563 5561 cpu_manager.go:414] "containerMap: removing container" podUID="6faf63d9-03e2-4f8e-8a04-92d2f6903e74" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706570 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6faf63d9-03e2-4f8e-8a04-92d2f6903e74" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706586 5561 cpu_manager.go:414] "containerMap: removing container" podUID="8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706593 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706602 5561 cpu_manager.go:414] "containerMap: removing container" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerName="istio-proxy" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706609 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerName="istio-proxy" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706620 5561 cpu_manager.go:414] "containerMap: removing container" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706626 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706637 5561 cpu_manager.go:414] "containerMap: removing container" podUID="e8ab9f5e-0844-477c-ab31-58dfc7df6d7d" containerName="maas-api" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706644 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ab9f5e-0844-477c-ab31-58dfc7df6d7d" containerName="maas-api" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706653 5561 cpu_manager.go:414] "containerMap: removing container" podUID="1797345c-e27c-423d-8b0c-71270e96647f" containerName="payload-processing" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706660 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1797345c-e27c-423d-8b0c-71270e96647f" containerName="payload-processing" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706673 5561 cpu_manager.go:414] "containerMap: removing container" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706680 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706690 5561 cpu_manager.go:414] "containerMap: removing container" podUID="0db8b320-9073-44fd-b41e-cb96b8e76fcf" containerName="maas-api" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706697 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db8b320-9073-44fd-b41e-cb96b8e76fcf" containerName="maas-api" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706799 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94eb3e8-cfd0-46ca-8b24-6746f4437e4e" containerName="maas-api" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706810 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa33f7d5-310c-4581-84c4-317226e57817" containerName="istio-proxy" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706820 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="de249c08-26bd-4582-b4c6-dce05a1ae3b5" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706828 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e6bd1e-cd4e-45a4-a9da-9003f53f359e" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706837 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f198ce3-2bfb-4b2d-bda5-cd81ff90e37d" containerName="maas-api" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706844 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="6faf63d9-03e2-4f8e-8a04-92d2f6903e74" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706854 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf23a0b-8cef-49dc-87f8-68ad4f0547b1" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706861 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a80651-6634-491d-89cb-59939a3c7ef8" containerName="istio-proxy" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706868 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b6de3b-682d-4a07-9297-e67df9f94243" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706892 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ab9f5e-0844-477c-ab31-58dfc7df6d7d" containerName="maas-api" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706899 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="1797345c-e27c-423d-8b0c-71270e96647f" containerName="payload-processing" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706906 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="0475c8c4-b045-4faf-b21f-6501ce60a3df" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706916 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db8b320-9073-44fd-b41e-cb96b8e76fcf" containerName="maas-api" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706930 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="05959640-ae04-4597-b190-232c13f303c2" containerName="maas-api" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706939 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="39722652-3460-46ed-89ce-5ab8a3eb255d" containerName="payload-pre-processing" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706948 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cfe3fa-bddf-4613-8987-9ebc09c064b4" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706958 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="46dbd4e4-a31e-4587-90f2-8bb3ad0832aa" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706967 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90f9ab7-7575-4bd8-b55b-93872c815747" containerName="payload-pre-processing" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706976 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="20cd41d9-5f2d-4ece-a7f7-e62487149e0a" containerName="main" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706988 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0ee610-89f1-4e6f-9017-1d3777be0b29" containerName="payload-processing" Nov 02 20:44:59.707700 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.706996 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b219f6-3577-4242-9bca-448a02c6b524" containerName="main" Nov 02 20:44:59.710234 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.710212 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.743249 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.743218 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:59.759699 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.759673 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fdd8j\"" Nov 02 20:44:59.759865 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.759815 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 20:44:59.808270 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.808208 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th"] Nov 02 20:44:59.879831 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.879745 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175f6a16-08f6-4d78-af68-b64b89b9d26a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.879831 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.879802 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175f6a16-08f6-4d78-af68-b64b89b9d26a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.880059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.879893 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftj2\" (UniqueName: \"kubernetes.io/projected/175f6a16-08f6-4d78-af68-b64b89b9d26a-kube-api-access-tftj2\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.880059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.879960 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.880059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.879994 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175f6a16-08f6-4d78-af68-b64b89b9d26a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.880059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.880013 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.880059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.880045 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.880278 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.880068 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.880278 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.880113 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.980833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.980782 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.980833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.980833 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.981134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.980865 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175f6a16-08f6-4d78-af68-b64b89b9d26a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.981134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.980920 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175f6a16-08f6-4d78-af68-b64b89b9d26a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.981134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.980939 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftj2\" (UniqueName: \"kubernetes.io/projected/175f6a16-08f6-4d78-af68-b64b89b9d26a-kube-api-access-tftj2\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.981134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.980971 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.981134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.981008 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175f6a16-08f6-4d78-af68-b64b89b9d26a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.981134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.981025 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.981134 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.981043 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.981500 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.981276 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.981500 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.981442 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.981500 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.981462 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.981622 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.981596 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.981756 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.981730 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175f6a16-08f6-4d78-af68-b64b89b9d26a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.983258 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.983236 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175f6a16-08f6-4d78-af68-b64b89b9d26a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:44:59.983477 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:44:59.983460 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175f6a16-08f6-4d78-af68-b64b89b9d26a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:45:00.035360 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:00.035321 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175f6a16-08f6-4d78-af68-b64b89b9d26a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:45:00.035533 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:00.035396 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftj2\" (UniqueName: \"kubernetes.io/projected/175f6a16-08f6-4d78-af68-b64b89b9d26a-kube-api-access-tftj2\") pod \"maas-default-gateway-openshift-default-696f47c6c-zc6th\" (UID: \"175f6a16-08f6-4d78-af68-b64b89b9d26a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:45:00.322694 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:00.322655 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:45:00.738726 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:00.738691 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:00.745611 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:00.745582 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th"] Nov 02 20:45:00.754474 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:45:00.754447 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175f6a16_08f6_4d78_af68_b64b89b9d26a.slice/crio-6ac09fb2516b70e7729fd7c4a48db91c436e7a5df0db6d7f88f2b2a2e7e84d1d WatchSource:0}: Error finding container 6ac09fb2516b70e7729fd7c4a48db91c436e7a5df0db6d7f88f2b2a2e7e84d1d: Status 404 returned error can't find the container with id 6ac09fb2516b70e7729fd7c4a48db91c436e7a5df0db6d7f88f2b2a2e7e84d1d Nov 02 20:45:00.756518 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:00.756484 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:45:00.756609 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:00.756553 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:45:00.756609 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:00.756586 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:45:01.757601 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:01.757565 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" event={"ID":"175f6a16-08f6-4d78-af68-b64b89b9d26a","Type":"ContainerStarted","Data":"814af43b73f0a11e81dffd915040a93dc6c75699888d3b30c6935275eea72e3c"} Nov 02 20:45:01.758078 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:01.757614 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" event={"ID":"175f6a16-08f6-4d78-af68-b64b89b9d26a","Type":"ContainerStarted","Data":"6ac09fb2516b70e7729fd7c4a48db91c436e7a5df0db6d7f88f2b2a2e7e84d1d"} Nov 02 20:45:01.918297 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:01.918240 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" podStartSLOduration=2.918225593 podStartE2EDuration="2.918225593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:01.916958646 +0000 UTC m=+3138.724742877" watchObservedRunningTime="2026-11-02 20:45:01.918225593 +0000 UTC m=+3138.726009810" Nov 02 20:45:02.323551 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:02.323512 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:45:02.324969 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:02.324934 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zc6th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:45:02.325091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:02.324998 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" podUID="175f6a16-08f6-4d78-af68-b64b89b9d26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 20:45:03.323990 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:03.323953 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zc6th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:45:03.324485 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:03.324014 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" podUID="175f6a16-08f6-4d78-af68-b64b89b9d26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 20:45:03.778892 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:45:03.778843 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:45:04.323927 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:04.323869 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zc6th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:45:04.324095 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:04.323953 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" podUID="175f6a16-08f6-4d78-af68-b64b89b9d26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 20:45:05.323501 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:05.323467 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zc6th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:45:05.323684 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:05.323521 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" podUID="175f6a16-08f6-4d78-af68-b64b89b9d26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 20:45:06.323590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:06.323552 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zc6th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:45:06.324040 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:06.323607 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" podUID="175f6a16-08f6-4d78-af68-b64b89b9d26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 20:45:07.323044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:07.322998 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zc6th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:45:07.323044 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:07.323046 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" podUID="175f6a16-08f6-4d78-af68-b64b89b9d26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 20:45:09.024412 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:09.024373 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:45:09.025141 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:09.025117 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:45:09.025867 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:09.025847 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" Nov 02 20:45:33.778222 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:45:33.778177 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:45:59.041112 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:59.041078 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:59.487706 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:45:59.487666 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:03.778633 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:46:03.778592 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:46:03.899182 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:03.899146 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:11.581298 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:11.581259 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:12.123359 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:12.123325 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:30.340591 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.340558 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx"] Nov 02 20:46:30.343707 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.343689 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.399565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.399534 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.399565 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.399565 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.399747 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.399589 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.399837 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.399783 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.399913 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.399893 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.399985 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.399942 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.400109 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.400087 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.400185 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.400170 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.400221 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.400197 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfl9b\" (UniqueName: \"kubernetes.io/projected/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-kube-api-access-sfl9b\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.442416 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.442381 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx"] Nov 02 20:46:30.501213 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.501181 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.501213 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.501217 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.501448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.501240 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.501448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.501273 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.501448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.501306 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfl9b\" (UniqueName: \"kubernetes.io/projected/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-kube-api-access-sfl9b\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.501448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.501336 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.501448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.501359 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.501448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.501389 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.501448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.501432 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.501889 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.501706 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.501889 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.501770 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.501889 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.501806 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.502066 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.502034 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.502125 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.502095 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.503529 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.503510 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.503751 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.503734 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.555628 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.555601 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.555765 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.555741 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfl9b\" (UniqueName: \"kubernetes.io/projected/def3f73d-a0f0-48ff-8d35-9bcb21cfecad-kube-api-access-sfl9b\") pod \"maas-default-gateway-openshift-default-696f47c6c-472vx\" (UID: \"def3f73d-a0f0-48ff-8d35-9bcb21cfecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:30.655642 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:30.655616 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:31.077807 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:31.077724 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx"] Nov 02 20:46:31.086305 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:46:31.086276 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef3f73d_a0f0_48ff_8d35_9bcb21cfecad.slice/crio-11e792f1aa351999dcbc1a4bb4bfdeb40cc11c2a834bcc88852ec0ae308b5dc2 WatchSource:0}: Error finding container 11e792f1aa351999dcbc1a4bb4bfdeb40cc11c2a834bcc88852ec0ae308b5dc2: Status 404 returned error can't find the container with id 11e792f1aa351999dcbc1a4bb4bfdeb40cc11c2a834bcc88852ec0ae308b5dc2 Nov 02 20:46:31.088480 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:31.088450 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:46:31.088561 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:31.088532 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:46:31.088610 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:31.088564 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:46:31.127081 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:31.126719 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" event={"ID":"def3f73d-a0f0-48ff-8d35-9bcb21cfecad","Type":"ContainerStarted","Data":"11e792f1aa351999dcbc1a4bb4bfdeb40cc11c2a834bcc88852ec0ae308b5dc2"} Nov 02 20:46:32.155238 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:32.155202 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" event={"ID":"def3f73d-a0f0-48ff-8d35-9bcb21cfecad","Type":"ContainerStarted","Data":"09d02ea634e44d2ed9335112bbaebeba7edc443e92e4325588d24dfa3bab86b8"} Nov 02 20:46:32.314748 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:32.314695 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" podStartSLOduration=2.314682045 podStartE2EDuration="2.314682045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:32.313891871 +0000 UTC m=+3229.121676082" watchObservedRunningTime="2026-11-02 20:46:32.314682045 +0000 UTC m=+3229.122466260" Nov 02 20:46:32.656079 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:32.656041 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:32.657708 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:32.657677 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-472vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:46:32.657858 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:32.657748 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" podUID="def3f73d-a0f0-48ff-8d35-9bcb21cfecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 20:46:33.656865 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:33.656827 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-472vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:46:33.657287 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:33.656900 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" podUID="def3f73d-a0f0-48ff-8d35-9bcb21cfecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 20:46:33.778823 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:46:33.778788 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:46:34.656532 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:34.656498 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-472vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:46:34.656722 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:34.656585 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" podUID="def3f73d-a0f0-48ff-8d35-9bcb21cfecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 20:46:35.657024 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:35.656987 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-472vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:46:35.657435 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:35.657042 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" podUID="def3f73d-a0f0-48ff-8d35-9bcb21cfecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 20:46:36.656963 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:36.656910 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-472vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:46:36.657132 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:36.656978 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" podUID="def3f73d-a0f0-48ff-8d35-9bcb21cfecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 20:46:37.656086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:37.656043 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-472vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:46:37.656315 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:37.656098 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" podUID="def3f73d-a0f0-48ff-8d35-9bcb21cfecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 20:46:39.339547 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:39.339514 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:39.340106 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:39.340067 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:39.340766 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:39.340742 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" Nov 02 20:46:59.358319 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:46:59.358212 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:00.579035 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.578995 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w"] Nov 02 20:47:00.584029 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.583999 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.605512 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.605485 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.605639 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.605524 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.605639 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.605546 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.605767 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.605635 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.605767 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.605677 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.605767 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.605727 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trj9\" (UniqueName: \"kubernetes.io/projected/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-kube-api-access-4trj9\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.605932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.605766 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.605932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.605834 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.605932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.605910 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.697229 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.697188 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w"] Nov 02 20:47:00.706496 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.706464 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.706496 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.706502 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.706758 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.706518 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.706758 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.706664 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.706758 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.706711 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.706965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.706756 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trj9\" (UniqueName: \"kubernetes.io/projected/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-kube-api-access-4trj9\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.706965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.706795 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.706965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.706819 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.706965 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.706924 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.707191 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.707048 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.707273 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.707252 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.707378 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.707332 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.707444 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.707389 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.707505 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.707443 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.709480 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.709452 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.709755 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.709732 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.761293 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.761261 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.761668 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.761640 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trj9\" (UniqueName: \"kubernetes.io/projected/6ab1cb1e-6c0c-494a-9736-fdbb755518e4-kube-api-access-4trj9\") pod \"maas-default-gateway-openshift-default-696f47c6c-48g6w\" (UID: \"6ab1cb1e-6c0c-494a-9736-fdbb755518e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:00.903815 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:00.903786 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:01.344863 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:01.344783 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w"] Nov 02 20:47:01.352731 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:47:01.352703 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab1cb1e_6c0c_494a_9736_fdbb755518e4.slice/crio-8ec982b3450c43bfc2cdb4e61c337ccb7270bb8f182d07f8c831722a80a18b6e WatchSource:0}: Error finding container 8ec982b3450c43bfc2cdb4e61c337ccb7270bb8f182d07f8c831722a80a18b6e: Status 404 returned error can't find the container with id 8ec982b3450c43bfc2cdb4e61c337ccb7270bb8f182d07f8c831722a80a18b6e Nov 02 20:47:01.354791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:01.354757 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:47:01.354866 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:01.354834 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:47:01.354927 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:01.354863 5561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 20:47:02.287120 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:02.287081 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" event={"ID":"6ab1cb1e-6c0c-494a-9736-fdbb755518e4","Type":"ContainerStarted","Data":"806801cff4b1ee030a497fa4f719cab5e1e7bf789e3385ed9d1d64c97ae5f838"} Nov 02 20:47:02.287552 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:02.287140 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" event={"ID":"6ab1cb1e-6c0c-494a-9736-fdbb755518e4","Type":"ContainerStarted","Data":"8ec982b3450c43bfc2cdb4e61c337ccb7270bb8f182d07f8c831722a80a18b6e"} Nov 02 20:47:02.904509 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:02.904473 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:02.905972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:02.905943 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-48g6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 20:47:02.906091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:02.905994 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" podUID="6ab1cb1e-6c0c-494a-9736-fdbb755518e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 20:47:03.779121 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:47:03.779082 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:47:03.904559 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:03.904519 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-48g6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 20:47:03.904744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:03.904574 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" podUID="6ab1cb1e-6c0c-494a-9736-fdbb755518e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 20:47:04.905151 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:04.905111 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-48g6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 20:47:04.905674 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:04.905164 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" podUID="6ab1cb1e-6c0c-494a-9736-fdbb755518e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 20:47:05.904414 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:05.904383 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-48g6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 20:47:05.904603 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:05.904448 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" podUID="6ab1cb1e-6c0c-494a-9736-fdbb755518e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 20:47:06.910159 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:06.909984 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-48g6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 20:47:06.910159 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:06.910055 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" podUID="6ab1cb1e-6c0c-494a-9736-fdbb755518e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 20:47:07.904289 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:07.904251 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-48g6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 20:47:07.904509 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:07.904317 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" podUID="6ab1cb1e-6c0c-494a-9736-fdbb755518e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 20:47:08.904657 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:08.904618 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-48g6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 20:47:08.905204 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:08.904675 5561 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" podUID="6ab1cb1e-6c0c-494a-9736-fdbb755518e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 20:47:09.908499 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:09.908466 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:09.908932 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:09.908752 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:09.909384 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:09.909364 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" Nov 02 20:47:10.068428 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:10.068375 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-48g6w" podStartSLOduration=10.068362864 podStartE2EDuration="10.068362864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:02.493950055 +0000 UTC m=+3259.301734270" watchObservedRunningTime="2026-11-02 20:47:10.068362864 +0000 UTC m=+3266.876147077" Nov 02 20:47:33.778341 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:47:33.778302 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:47:59.609898 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:47:59.609838 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:00.331606 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:48:00.331556 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:03.777996 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:48:03.777955 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:48:33.777376 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:48:33.777337 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:49:00.104500 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:49:00.104446 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:00.648091 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:49:00.648013 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:03.778345 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:49:03.778300 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:49:10.608113 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:49:10.608057 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:33.779185 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:49:33.779141 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:50:00.357636 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:50:00.357600 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:50:03.778593 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:50:03.778557 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:50:33.779750 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:50:33.779710 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:51:00.733702 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:51:00.733664 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:51:01.366571 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:51:01.366535 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:51:03.780041 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:51:03.779998 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:51:33.778596 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:51:33.778552 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:51:55.077080 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:51:55.077022 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zc6th container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:51:55.077833 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:51:55.077768 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" podUID="175f6a16-08f6-4d78-af68-b64b89b9d26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:51:55.385966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:51:55.385001 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-472vx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:51:55.385966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:51:55.385080 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" podUID="def3f73d-a0f0-48ff-8d35-9bcb21cfecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:52:01.049143 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:52:01.049107 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:52:01.648820 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:52:01.648783 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:52:03.783826 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:52:03.783786 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:52:23.372991 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:52:23.372949 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:52:33.819952 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:52:33.819914 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:52:49.063663 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:52:49.063621 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:52:49.661156 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:52:49.661112 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:52:50.001453 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:52:50.001373 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:52:54.007202 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:52:54.007115 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:52:54.223520 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:52:54.223483 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:52:57.196980 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:52:57.196948 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:52:57.482351 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:52:57.482261 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:52:59.929409 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:52:59.929377 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:00.100032 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:00.100000 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:00.383607 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:00.383498 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:01.364814 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:01.364762 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:03.779101 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:53:03.779067 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:53:04.763577 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:04.763538 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:09.360657 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:09.360619 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:13.858373 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:13.858330 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:18.709399 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:18.709361 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:18.886397 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:18.886360 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:19.480252 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:19.480215 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:19.701053 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:19.701015 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:22.398283 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.398244 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck"] Nov 02 20:53:22.401169 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.401144 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.435457 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.435421 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-dshm\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.435457 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.435477 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-model-cache\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.435731 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.435575 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tmp-dir\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.435731 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.435593 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tls-certs\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.435731 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.435655 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-home\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.435731 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.435675 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkj92\" (UniqueName: \"kubernetes.io/projected/21ea1352-e51e-48af-8006-9b9fe4a10ab4-kube-api-access-bkj92\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.452196 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.452166 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0a02f7\"/\"openshift-service-ca.crt\"" Nov 02 20:53:22.499743 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.499704 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0a02f7\"/\"kube-root-ca.crt\"" Nov 02 20:53:22.499984 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.499783 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0a02f7\"/\"default-dockercfg-lf7v7\"" Nov 02 20:53:22.499984 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.499867 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0a02f7\"/\"simulator-single-main-0a02f7-kserve-self-signed-certs\"" Nov 02 20:53:22.500916 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.500894 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck"] Nov 02 20:53:22.536213 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.536176 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-dshm\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.536374 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.536264 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-model-cache\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.536374 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.536324 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tmp-dir\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.536374 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.536348 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tls-certs\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.536554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.536384 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-home\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.536554 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.536415 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkj92\" (UniqueName: \"kubernetes.io/projected/21ea1352-e51e-48af-8006-9b9fe4a10ab4-kube-api-access-bkj92\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.536791 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.536738 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-model-cache\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.537747 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.537720 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-home\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.538059 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.538032 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tmp-dir\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.539545 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.539521 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tls-certs\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.540131 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.540110 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-dshm\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.543891 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.543827 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:22.590583 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.590545 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkj92\" (UniqueName: \"kubernetes.io/projected/21ea1352-e51e-48af-8006-9b9fe4a10ab4-kube-api-access-bkj92\") pod \"simulator-single-main-0a02f7-kserve-54f595b844-wvbck\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:22.712534 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:22.712487 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:23.089798 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:23.089711 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck"] Nov 02 20:53:23.097328 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:53:23.097302 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21ea1352_e51e_48af_8006_9b9fe4a10ab4.slice/crio-ee458690e689c235a4fedb72554470948cc34c8c3c2dabd29cb7d662fc155aa1 WatchSource:0}: Error finding container ee458690e689c235a4fedb72554470948cc34c8c3c2dabd29cb7d662fc155aa1: Status 404 returned error can't find the container with id ee458690e689c235a4fedb72554470948cc34c8c3c2dabd29cb7d662fc155aa1 Nov 02 20:53:23.350032 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:23.349933 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:24.106809 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:24.106766 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" event={"ID":"21ea1352-e51e-48af-8006-9b9fe4a10ab4","Type":"ContainerStarted","Data":"030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521"} Nov 02 20:53:24.106809 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:24.106814 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" event={"ID":"21ea1352-e51e-48af-8006-9b9fe4a10ab4","Type":"ContainerStarted","Data":"ee458690e689c235a4fedb72554470948cc34c8c3c2dabd29cb7d662fc155aa1"} Nov 02 20:53:24.216132 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:24.216073 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" podStartSLOduration=2.21606008 podStartE2EDuration="2.21606008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:53:24.2150695 +0000 UTC m=+3641.022853742" watchObservedRunningTime="2026-11-02 20:53:24.21606008 +0000 UTC m=+3641.023844297" Nov 02 20:53:24.713650 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:24.713611 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:28.005092 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:28.005056 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:32.475848 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:32.475807 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:32.713254 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:32.713216 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:32.725958 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:32.725849 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:33.154640 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:33.154619 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:53:33.778739 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:53:33.778699 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:53:37.054522 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:37.054475 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:41.552307 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:41.552275 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:46.149894 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:46.149833 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:50.675508 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:50.675475 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:55.255096 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:55.255062 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:59.775365 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:53:59.775320 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:01.759820 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:01.759779 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:02.327312 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:02.327274 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:03.778731 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:54:03.778693 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:54:04.346951 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:04.346913 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:08.855575 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:08.855534 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:13.454806 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:13.454761 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:17.968661 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:17.968628 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:22.545389 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:22.545348 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:27.148191 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:27.148106 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:28.622744 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:28.622704 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:31.952799 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:31.952768 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:33.532679 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:33.532646 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck"] Nov 02 20:54:33.649725 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:33.649692 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:33.779112 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:54:33.779071 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:54:36.473627 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:36.473587 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:40.483610 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:40.483576 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:40.951995 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:40.951945 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:41.121523 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:41.121484 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:44.924358 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:44.924307 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:45.196628 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:45.196538 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:45.913720 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:45.913684 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:48.552226 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.552187 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" podUID="21ea1352-e51e-48af-8006-9b9fe4a10ab4" containerName="main" containerID="cri-o://030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521" gracePeriod=45 Nov 02 20:54:48.788772 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.788747 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:54:48.851037 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.850962 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tls-certs\" (UniqueName: \"kubernetes.io/secret/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tls-certs\") pod \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " Nov 02 20:54:48.851037 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.851003 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-model-cache\") pod \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " Nov 02 20:54:48.851037 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.851028 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tmp-dir\") pod \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " Nov 02 20:54:48.851329 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.851133 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ea1352-e51e-48af-8006-9b9fe4a10ab4-kube-api-access-bkj92\" (UniqueName: \"kubernetes.io/projected/21ea1352-e51e-48af-8006-9b9fe4a10ab4-kube-api-access-bkj92\") pod \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " Nov 02 20:54:48.851329 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.851216 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-home\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-home\") pod \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " Nov 02 20:54:48.851329 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.851253 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-model-cache" pod "21ea1352-e51e-48af-8006-9b9fe4a10ab4" (UID: "21ea1352-e51e-48af-8006-9b9fe4a10ab4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:48.851329 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.851276 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-dshm\") pod \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\" (UID: \"21ea1352-e51e-48af-8006-9b9fe4a10ab4\") " Nov 02 20:54:48.851329 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.851274 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tmp-dir" pod "21ea1352-e51e-48af-8006-9b9fe4a10ab4" (UID: "21ea1352-e51e-48af-8006-9b9fe4a10ab4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:48.851526 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.851443 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-home" pod "21ea1352-e51e-48af-8006-9b9fe4a10ab4" (UID: "21ea1352-e51e-48af-8006-9b9fe4a10ab4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:48.851580 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.851562 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:48.851621 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.851585 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:48.851621 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.851598 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:48.853285 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.853245 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-dshm" pod "21ea1352-e51e-48af-8006-9b9fe4a10ab4" (UID: "21ea1352-e51e-48af-8006-9b9fe4a10ab4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:48.853381 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.853330 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ea1352-e51e-48af-8006-9b9fe4a10ab4-kube-api-access-bkj92" pod "21ea1352-e51e-48af-8006-9b9fe4a10ab4" (UID: "21ea1352-e51e-48af-8006-9b9fe4a10ab4"). InnerVolumeSpecName "kube-api-access-bkj92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:48.853381 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.853331 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tls-certs" pod "21ea1352-e51e-48af-8006-9b9fe4a10ab4" (UID: "21ea1352-e51e-48af-8006-9b9fe4a10ab4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:54:48.952715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.952687 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ea1352-e51e-48af-8006-9b9fe4a10ab4-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:48.952715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.952711 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ea1352-e51e-48af-8006-9b9fe4a10ab4-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:48.952715 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:48.952721 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkj92\" (UniqueName: \"kubernetes.io/projected/21ea1352-e51e-48af-8006-9b9fe4a10ab4-kube-api-access-bkj92\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:49.474321 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:49.474286 5561 generic.go:356] "Generic (PLEG): container finished" podID="21ea1352-e51e-48af-8006-9b9fe4a10ab4" containerID="030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521" exitCode=0 Nov 02 20:54:49.474489 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:49.474334 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" Nov 02 20:54:49.474489 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:49.474375 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" event={"ID":"21ea1352-e51e-48af-8006-9b9fe4a10ab4","Type":"ContainerDied","Data":"030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521"} Nov 02 20:54:49.474489 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:49.474425 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck" event={"ID":"21ea1352-e51e-48af-8006-9b9fe4a10ab4","Type":"ContainerDied","Data":"ee458690e689c235a4fedb72554470948cc34c8c3c2dabd29cb7d662fc155aa1"} Nov 02 20:54:49.474489 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:49.474446 5561 scope.go:122] "RemoveContainer" containerID="030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521" Nov 02 20:54:49.482743 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:49.482722 5561 scope.go:122] "RemoveContainer" containerID="030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521" Nov 02 20:54:49.483029 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:54:49.483011 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521\": container with ID starting with 030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521 not found: ID does not exist" containerID="030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521" Nov 02 20:54:49.483087 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:49.483037 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521"} err="failed to get container status \"030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521\": rpc error: code = NotFound desc = could not find container \"030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521\": container with ID starting with 030f0cfe1eef1ddc0e743d8c38c0ae66587b0071564bf202f4ac2bfe6e900521 not found: ID does not exist" Nov 02 20:54:49.629821 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:49.629728 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck"] Nov 02 20:54:49.633014 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:49.632989 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-0a02f7/simulator-single-main-0a02f7-kserve-54f595b844-wvbck"] Nov 02 20:54:51.311923 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:51.311866 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ea1352-e51e-48af-8006-9b9fe4a10ab4" path="/var/lib/kubelet/pods/21ea1352-e51e-48af-8006-9b9fe4a10ab4/volumes" Nov 02 20:54:51.630487 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:51.630408 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:51.895435 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:51.895346 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:52.095703 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:52.095670 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:57.984155 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:54:57.984115 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:55:02.002172 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:55:02.002129 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:55:02.647025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:55:02.646989 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:55:03.779480 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:55:03.779442 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:55:09.796825 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:55:09.783036 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:55:10.094160 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:55:10.094020 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zc6th container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:55:10.094475 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:55:10.094426 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zc6th" podUID="175f6a16-08f6-4d78-af68-b64b89b9d26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:55:10.386101 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:55:10.386000 5561 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-472vx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:55:10.386381 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:55:10.386349 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-472vx" podUID="def3f73d-a0f0-48ff-8d35-9bcb21cfecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:55:10.495903 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:55:10.494468 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:55:15.942136 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:55:15.942103 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:55:28.987219 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:55:28.987179 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:55:29.170049 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:55:29.170013 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:55:33.779462 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:55:33.779423 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:56:02.324914 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:56:02.324803 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:56:03.777901 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:56:03.777835 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:56:33.779263 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:56:33.779225 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:57:02.649019 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:02.648979 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:03.333069 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:03.333027 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:03.778792 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:57:03.778760 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:57:29.499751 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:29.499663 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:33.778715 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:57:33.778674 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:57:37.378534 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:37.378497 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:38.095405 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:38.095369 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:38.373204 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:38.373119 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:42.634023 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:42.633987 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:42.869402 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:42.869370 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:43.058085 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:43.058054 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:45.926481 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:45.926445 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:46.105477 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:46.105438 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:48.718141 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:48.718103 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:48.897506 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:48.897442 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:49.075761 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:49.075681 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:53.364456 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:53.364415 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:57:57.981315 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:57:57.981271 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:02.567548 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:02.567509 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:02.981428 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:02.981394 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:03.666132 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:03.666088 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:03.779774 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:58:03.779736 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:58:07.498268 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:07.498216 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:07.675461 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:07.675426 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:08.061252 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:08.061216 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:08.245717 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:08.245687 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:12.068436 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:12.068390 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:16.571367 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:16.571305 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:18.206564 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:18.206517 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:20.818970 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.818939 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2"] Nov 02 20:58:20.819400 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.819316 5561 cpu_manager.go:414] "containerMap: removing container" podUID="21ea1352-e51e-48af-8006-9b9fe4a10ab4" containerName="main" Nov 02 20:58:20.819400 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.819328 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ea1352-e51e-48af-8006-9b9fe4a10ab4" containerName="main" Nov 02 20:58:20.819490 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.819442 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ea1352-e51e-48af-8006-9b9fe4a10ab4" containerName="main" Nov 02 20:58:20.822472 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.822453 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.888391 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.888359 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-tmp-dir\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.888590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.888411 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfts\" (UniqueName: \"kubernetes.io/projected/5de3039f-65cc-402a-a494-276af15af289-kube-api-access-nhfts\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.888590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.888524 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-model-cache\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.888590 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.888573 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-home\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.888777 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.888596 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-dshm\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.888777 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.888685 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de3039f-65cc-402a-a494-276af15af289-tls-certs\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.921630 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.921595 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d512fb\"/\"openshift-service-ca.crt\"" Nov 02 20:58:20.921630 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.921598 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d512fb\"/\"default-dockercfg-x5h9z\"" Nov 02 20:58:20.921918 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.921597 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d512fb\"/\"kube-root-ca.crt\"" Nov 02 20:58:20.921918 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.921733 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d512fb\"/\"simulator-single-main-d512fb-kserve-self-signed-certs\"" Nov 02 20:58:20.922699 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.922680 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2"] Nov 02 20:58:20.963539 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.963490 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:20.989278 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.989246 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfts\" (UniqueName: \"kubernetes.io/projected/5de3039f-65cc-402a-a494-276af15af289-kube-api-access-nhfts\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.989466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.989296 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-model-cache\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.989466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.989329 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-home\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.989466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.989351 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-dshm\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.989466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.989390 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de3039f-65cc-402a-a494-276af15af289-tls-certs\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.989466 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.989413 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-tmp-dir\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.989756 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.989736 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-model-cache\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.989831 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.989803 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-home\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.989915 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.989831 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-tmp-dir\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.991798 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.991776 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-dshm\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:20.991972 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:20.991955 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de3039f-65cc-402a-a494-276af15af289-tls-certs\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:21.044001 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:21.043961 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfts\" (UniqueName: \"kubernetes.io/projected/5de3039f-65cc-402a-a494-276af15af289-kube-api-access-nhfts\") pod \"simulator-single-main-d512fb-kserve-598c94bbb6-95ff2\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:21.129502 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:21.129420 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:21.134401 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:21.134368 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:21.510279 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:21.510247 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2"] Nov 02 20:58:21.518519 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:58:21.518490 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5de3039f_65cc_402a_a494_276af15af289.slice/crio-c9443c3210b4ffcae7e8e5982bb5bece1c6e01d4bc29b95b8832e6154e247f61 WatchSource:0}: Error finding container c9443c3210b4ffcae7e8e5982bb5bece1c6e01d4bc29b95b8832e6154e247f61: Status 404 returned error can't find the container with id c9443c3210b4ffcae7e8e5982bb5bece1c6e01d4bc29b95b8832e6154e247f61 Nov 02 20:58:22.458448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:22.458410 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" event={"ID":"5de3039f-65cc-402a-a494-276af15af289","Type":"ContainerStarted","Data":"966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe"} Nov 02 20:58:22.458448 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:22.458454 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" event={"ID":"5de3039f-65cc-402a-a494-276af15af289","Type":"ContainerStarted","Data":"c9443c3210b4ffcae7e8e5982bb5bece1c6e01d4bc29b95b8832e6154e247f61"} Nov 02 20:58:22.566892 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:22.566821 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" podStartSLOduration=2.566808675 podStartE2EDuration="2.566808675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:58:22.566533104 +0000 UTC m=+3939.374317339" watchObservedRunningTime="2026-11-02 20:58:22.566808675 +0000 UTC m=+3939.374592890" Nov 02 20:58:23.135011 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:23.134972 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:25.769412 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:25.769370 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:30.371734 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:30.371699 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:31.134638 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:31.134600 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:31.146940 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:31.146913 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:31.508960 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:31.508928 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:58:33.778532 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:58:33.778494 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:58:34.982802 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:34.982763 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:39.461410 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:39.461375 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:44.078971 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:44.078924 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:48.615113 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:48.615075 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:53.164259 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:53.164227 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:55.975989 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:55.975904 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:56.150496 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:56.150462 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:57.653457 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:58:57.653422 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:02.228211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:02.228171 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:03.338027 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:03.337990 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:03.778274 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:59:03.778237 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:59:06.657366 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:06.657313 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:09.068797 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:09.068755 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:09.243799 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:09.243762 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:11.261678 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:11.261629 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:15.756047 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:15.756005 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:17.292086 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:17.292051 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:20.557257 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:20.557218 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:22.148483 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:22.148452 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2"] Nov 02 20:59:22.261363 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:22.261292 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:25.365651 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:25.365610 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:29.353733 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:29.353696 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:29.590803 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:29.590767 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:29.757728 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:29.757691 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:33.691218 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:33.691181 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:33.778923 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:59:33.778860 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 20:59:33.971268 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:33.971163 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:34.750920 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:34.750868 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:37.168601 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.168555 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" podUID="5de3039f-65cc-402a-a494-276af15af289" containerName="main" containerID="cri-o://966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe" gracePeriod=45 Nov 02 20:59:37.413935 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.413904 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:59:37.503540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.503445 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-tmp-dir\") pod \"5de3039f-65cc-402a-a494-276af15af289\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " Nov 02 20:59:37.503540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.503480 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-dshm\") pod \"5de3039f-65cc-402a-a494-276af15af289\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " Nov 02 20:59:37.503540 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.503543 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-model-cache\") pod \"5de3039f-65cc-402a-a494-276af15af289\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " Nov 02 20:59:37.503836 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.503574 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-home\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-home\") pod \"5de3039f-65cc-402a-a494-276af15af289\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " Nov 02 20:59:37.503836 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.503635 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de3039f-65cc-402a-a494-276af15af289-kube-api-access-nhfts\" (UniqueName: \"kubernetes.io/projected/5de3039f-65cc-402a-a494-276af15af289-kube-api-access-nhfts\") pod \"5de3039f-65cc-402a-a494-276af15af289\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " Nov 02 20:59:37.503836 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.503655 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de3039f-65cc-402a-a494-276af15af289-tls-certs\" (UniqueName: \"kubernetes.io/secret/5de3039f-65cc-402a-a494-276af15af289-tls-certs\") pod \"5de3039f-65cc-402a-a494-276af15af289\" (UID: \"5de3039f-65cc-402a-a494-276af15af289\") " Nov 02 20:59:37.503836 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.503772 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-tmp-dir" pod "5de3039f-65cc-402a-a494-276af15af289" (UID: "5de3039f-65cc-402a-a494-276af15af289"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:37.504025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.503846 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-home" pod "5de3039f-65cc-402a-a494-276af15af289" (UID: "5de3039f-65cc-402a-a494-276af15af289"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:37.504025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.503860 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-model-cache" pod "5de3039f-65cc-402a-a494-276af15af289" (UID: "5de3039f-65cc-402a-a494-276af15af289"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:37.504025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.503994 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:37.504025 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.504014 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:37.504214 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.504026 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:37.505916 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.505895 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de3039f-65cc-402a-a494-276af15af289-kube-api-access-nhfts" pod "5de3039f-65cc-402a-a494-276af15af289" (UID: "5de3039f-65cc-402a-a494-276af15af289"). InnerVolumeSpecName "kube-api-access-nhfts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:59:37.506007 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.505936 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-dshm" pod "5de3039f-65cc-402a-a494-276af15af289" (UID: "5de3039f-65cc-402a-a494-276af15af289"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:37.506007 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.505951 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de3039f-65cc-402a-a494-276af15af289-tls-certs" pod "5de3039f-65cc-402a-a494-276af15af289" (UID: "5de3039f-65cc-402a-a494-276af15af289"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:59:37.604947 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.604913 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhfts\" (UniqueName: \"kubernetes.io/projected/5de3039f-65cc-402a-a494-276af15af289-kube-api-access-nhfts\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:37.604947 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.604945 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de3039f-65cc-402a-a494-276af15af289-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:37.604947 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.604954 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5de3039f-65cc-402a-a494-276af15af289-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:37.797617 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.797534 5561 generic.go:356] "Generic (PLEG): container finished" podID="5de3039f-65cc-402a-a494-276af15af289" containerID="966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe" exitCode=0 Nov 02 20:59:37.797617 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.797575 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" Nov 02 20:59:37.797974 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.797615 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" event={"ID":"5de3039f-65cc-402a-a494-276af15af289","Type":"ContainerDied","Data":"966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe"} Nov 02 20:59:37.797974 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.797654 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2" event={"ID":"5de3039f-65cc-402a-a494-276af15af289","Type":"ContainerDied","Data":"c9443c3210b4ffcae7e8e5982bb5bece1c6e01d4bc29b95b8832e6154e247f61"} Nov 02 20:59:37.797974 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.797671 5561 scope.go:122] "RemoveContainer" containerID="966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe" Nov 02 20:59:37.809606 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.809584 5561 scope.go:122] "RemoveContainer" containerID="966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe" Nov 02 20:59:37.809902 ip-10-0-134-194 kubenswrapper[5561]: E1102 20:59:37.809866 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe\": container with ID starting with 966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe not found: ID does not exist" containerID="966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe" Nov 02 20:59:37.809967 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.809912 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe"} err="failed to get container status \"966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe\": rpc error: code = NotFound desc = could not find container \"966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe\": container with ID starting with 966e143c62b0951e46e92394c1f1845547212fdf2901df63f73e5f77e3eefefe not found: ID does not exist" Nov 02 20:59:37.950856 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.950820 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2"] Nov 02 20:59:37.998401 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:37.998371 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-d512fb/simulator-single-main-d512fb-kserve-598c94bbb6-95ff2"] Nov 02 20:59:39.312195 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:39.312165 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de3039f-65cc-402a-a494-276af15af289" path="/var/lib/kubelet/pods/5de3039f-65cc-402a-a494-276af15af289/volumes" Nov 02 20:59:40.476432 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:40.476392 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:40.658161 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:40.658125 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:40.888359 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:40.888284 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:46.656688 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.656651 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89"] Nov 02 20:59:46.657223 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.657164 5561 cpu_manager.go:414] "containerMap: removing container" podUID="5de3039f-65cc-402a-a494-276af15af289" containerName="main" Nov 02 20:59:46.657223 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.657182 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de3039f-65cc-402a-a494-276af15af289" containerName="main" Nov 02 20:59:46.657346 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.657306 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de3039f-65cc-402a-a494-276af15af289" containerName="main" Nov 02 20:59:46.661788 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.661767 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.688761 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.688732 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hblv7\" (UniqueName: \"kubernetes.io/projected/e2152792-5adb-40d3-b8bf-b8c5554d3338-kube-api-access-hblv7\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.688935 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.688848 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-model-cache\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.688935 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.688926 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-dshm\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.689063 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.688971 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-home\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.689063 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.688996 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-tmp-dir\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.689163 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.689061 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2152792-5adb-40d3-b8bf-b8c5554d3338-tls-certs\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.712281 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.712250 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89"] Nov 02 20:59:46.758268 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.758241 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2c9f5f8f-kserve-self-signed-certs\"" Nov 02 20:59:46.789966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.789929 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-home\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.789966 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.789966 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-tmp-dir\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.790207 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.790004 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2152792-5adb-40d3-b8bf-b8c5554d3338-tls-certs\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.790207 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.790055 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hblv7\" (UniqueName: \"kubernetes.io/projected/e2152792-5adb-40d3-b8bf-b8c5554d3338-kube-api-access-hblv7\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.790207 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.790106 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-model-cache\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.790460 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.790380 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-dshm\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.790460 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.790412 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-tmp-dir\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.790636 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.790389 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-home\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.790636 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.790499 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-model-cache\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.790974 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.790949 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:46.792512 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.792492 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-dshm\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.792635 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.792616 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2152792-5adb-40d3-b8bf-b8c5554d3338-tls-certs\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.844084 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.844053 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hblv7\" (UniqueName: \"kubernetes.io/projected/e2152792-5adb-40d3-b8bf-b8c5554d3338-kube-api-access-hblv7\") pod \"e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:46.973741 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:46.973648 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:47.373755 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:47.373681 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89"] Nov 02 20:59:47.381657 ip-10-0-134-194 kubenswrapper[5561]: W1102 20:59:47.381615 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2152792_5adb_40d3_b8bf_b8c5554d3338.slice/crio-4f8f493570aaca7af044ab4de63b20e7d4181d7b543b424235807afecc290937 WatchSource:0}: Error finding container 4f8f493570aaca7af044ab4de63b20e7d4181d7b543b424235807afecc290937: Status 404 returned error can't find the container with id 4f8f493570aaca7af044ab4de63b20e7d4181d7b543b424235807afecc290937 Nov 02 20:59:47.842373 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:47.842341 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" event={"ID":"e2152792-5adb-40d3-b8bf-b8c5554d3338","Type":"ContainerStarted","Data":"ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3"} Nov 02 20:59:47.842907 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:47.842383 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" event={"ID":"e2152792-5adb-40d3-b8bf-b8c5554d3338","Type":"ContainerStarted","Data":"4f8f493570aaca7af044ab4de63b20e7d4181d7b543b424235807afecc290937"} Nov 02 20:59:47.950527 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:47.950473 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" podStartSLOduration=1.950459382 podStartE2EDuration="1.950459382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:59:47.950397148 +0000 UTC m=+4024.758181382" watchObservedRunningTime="2026-11-02 20:59:47.950459382 +0000 UTC m=+4024.758243598" Nov 02 20:59:47.974230 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:47.974199 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:56.974843 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:56.974803 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:56.987256 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:56.987224 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:57.912135 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:57.912102 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 20:59:59.416211 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:59.416145 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:59.632295 ip-10-0-134-194 kubenswrapper[5561]: I1102 20:59:59.632264 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:00:03.647318 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:03.647277 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:00:03.779747 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:00:03.779704 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:00:04.347897 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:04.347836 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:00:11.266037 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:11.265986 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:00:13.588991 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:13.588948 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:00:30.161318 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.161284 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6"] Nov 02 21:00:30.164262 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.164244 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.219893 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.219848 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6"] Nov 02 21:00:30.238148 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.238107 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-dshm\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.238346 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.238160 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c270f91a-06a9-4988-9afa-68fa72b41261-tls-certs\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.238346 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.238261 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvwm\" (UniqueName: \"kubernetes.io/projected/c270f91a-06a9-4988-9afa-68fa72b41261-kube-api-access-dhvwm\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.238346 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.238328 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-tmp-dir\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.238531 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.238398 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-model-cache\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.238531 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.238427 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-home\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.265034 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.265002 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2c9f5f8f-kserve-self-signed-certs\"" Nov 02 21:00:30.300420 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.300385 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:00:30.339054 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.339020 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-model-cache\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.339248 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.339066 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-home\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.339248 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.339122 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-dshm\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.339248 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.339140 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c270f91a-06a9-4988-9afa-68fa72b41261-tls-certs\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.339248 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.339167 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvwm\" (UniqueName: \"kubernetes.io/projected/c270f91a-06a9-4988-9afa-68fa72b41261-kube-api-access-dhvwm\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.339248 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.339217 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-tmp-dir\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.339538 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.339496 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-model-cache\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.339596 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.339535 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-home\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.339700 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.339673 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-tmp-dir\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.341533 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.341510 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-dshm\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.341823 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.341793 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c270f91a-06a9-4988-9afa-68fa72b41261-tls-certs\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.394493 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.394456 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvwm\" (UniqueName: \"kubernetes.io/projected/c270f91a-06a9-4988-9afa-68fa72b41261-kube-api-access-dhvwm\") pod \"e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.476412 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.476305 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:30.844973 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:30.844891 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6"] Nov 02 21:00:30.853972 ip-10-0-134-194 kubenswrapper[5561]: W1102 21:00:30.853932 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc270f91a_06a9_4988_9afa_68fa72b41261.slice/crio-efd1f6006e3841844ad1b63efe59bf847f45245c66bb0485e5b45cc0cfeae55e WatchSource:0}: Error finding container efd1f6006e3841844ad1b63efe59bf847f45245c66bb0485e5b45cc0cfeae55e: Status 404 returned error can't find the container with id efd1f6006e3841844ad1b63efe59bf847f45245c66bb0485e5b45cc0cfeae55e Nov 02 21:00:31.050639 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:31.050602 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" event={"ID":"c270f91a-06a9-4988-9afa-68fa72b41261","Type":"ContainerStarted","Data":"521389a394e2bae548c927809e89502b823c6e854a1b3650e6706884fcbcb53a"} Nov 02 21:00:31.050834 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:31.050650 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" event={"ID":"c270f91a-06a9-4988-9afa-68fa72b41261","Type":"ContainerStarted","Data":"efd1f6006e3841844ad1b63efe59bf847f45245c66bb0485e5b45cc0cfeae55e"} Nov 02 21:00:31.159228 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:31.159164 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" podStartSLOduration=1.159149602 podStartE2EDuration="1.159149602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:31.158634021 +0000 UTC m=+4067.966418257" watchObservedRunningTime="2026-11-02 21:00:31.159149602 +0000 UTC m=+4067.966933817" Nov 02 21:00:31.477211 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:31.477124 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:33.778432 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:00:33.778394 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:00:40.476972 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:40.476930 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:40.489714 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:40.489685 5561 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:00:41.126222 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:00:41.126187 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:01:03.781219 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:01:03.781171 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:01:04.180896 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:04.180853 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:01:04.667814 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:04.667774 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:01:11.793121 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:11.793080 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89"] Nov 02 21:01:11.886510 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:11.886465 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:01:20.246506 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:20.246467 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6"] Nov 02 21:01:20.306734 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:20.306674 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:01:26.812937 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:26.812892 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" podUID="e2152792-5adb-40d3-b8bf-b8c5554d3338" containerName="main" containerID="cri-o://ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3" gracePeriod=45 Nov 02 21:01:26.901782 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:26.901738 5561 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" podUID="e2152792-5adb-40d3-b8bf-b8c5554d3338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.106:8000/ready\": dial tcp 10.134.0.106:8000: connect: connection refused" Nov 02 21:01:27.057081 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.057054 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 21:01:27.114024 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.113941 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-home\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-home\") pod \"e2152792-5adb-40d3-b8bf-b8c5554d3338\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " Nov 02 21:01:27.114024 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.114000 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-dshm\") pod \"e2152792-5adb-40d3-b8bf-b8c5554d3338\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " Nov 02 21:01:27.114024 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.114023 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-tmp-dir\") pod \"e2152792-5adb-40d3-b8bf-b8c5554d3338\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " Nov 02 21:01:27.114305 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.114052 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-model-cache\") pod \"e2152792-5adb-40d3-b8bf-b8c5554d3338\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " Nov 02 21:01:27.114305 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.114094 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2152792-5adb-40d3-b8bf-b8c5554d3338-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2152792-5adb-40d3-b8bf-b8c5554d3338-tls-certs\") pod \"e2152792-5adb-40d3-b8bf-b8c5554d3338\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " Nov 02 21:01:27.114305 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.114118 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2152792-5adb-40d3-b8bf-b8c5554d3338-kube-api-access-hblv7\" (UniqueName: \"kubernetes.io/projected/e2152792-5adb-40d3-b8bf-b8c5554d3338-kube-api-access-hblv7\") pod \"e2152792-5adb-40d3-b8bf-b8c5554d3338\" (UID: \"e2152792-5adb-40d3-b8bf-b8c5554d3338\") " Nov 02 21:01:27.114419 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.114267 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-home" pod "e2152792-5adb-40d3-b8bf-b8c5554d3338" (UID: "e2152792-5adb-40d3-b8bf-b8c5554d3338"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:27.114419 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.114338 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-tmp-dir" pod "e2152792-5adb-40d3-b8bf-b8c5554d3338" (UID: "e2152792-5adb-40d3-b8bf-b8c5554d3338"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:27.114419 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.114355 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-model-cache" pod "e2152792-5adb-40d3-b8bf-b8c5554d3338" (UID: "e2152792-5adb-40d3-b8bf-b8c5554d3338"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:27.114541 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.114456 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:27.114541 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.114474 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:27.114541 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.114488 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:27.116457 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.116422 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2152792-5adb-40d3-b8bf-b8c5554d3338-tls-certs" pod "e2152792-5adb-40d3-b8bf-b8c5554d3338" (UID: "e2152792-5adb-40d3-b8bf-b8c5554d3338"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:01:27.116571 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.116456 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-dshm" pod "e2152792-5adb-40d3-b8bf-b8c5554d3338" (UID: "e2152792-5adb-40d3-b8bf-b8c5554d3338"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:27.116571 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.116537 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2152792-5adb-40d3-b8bf-b8c5554d3338-kube-api-access-hblv7" pod "e2152792-5adb-40d3-b8bf-b8c5554d3338" (UID: "e2152792-5adb-40d3-b8bf-b8c5554d3338"). InnerVolumeSpecName "kube-api-access-hblv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:27.215443 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.215408 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2152792-5adb-40d3-b8bf-b8c5554d3338-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:27.215443 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.215441 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hblv7\" (UniqueName: \"kubernetes.io/projected/e2152792-5adb-40d3-b8bf-b8c5554d3338-kube-api-access-hblv7\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:27.215443 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.215451 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2152792-5adb-40d3-b8bf-b8c5554d3338-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:27.372668 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.372589 5561 generic.go:356] "Generic (PLEG): container finished" podID="e2152792-5adb-40d3-b8bf-b8c5554d3338" containerID="ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3" exitCode=0 Nov 02 21:01:27.372668 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.372635 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" event={"ID":"e2152792-5adb-40d3-b8bf-b8c5554d3338","Type":"ContainerDied","Data":"ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3"} Nov 02 21:01:27.372668 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.372663 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" event={"ID":"e2152792-5adb-40d3-b8bf-b8c5554d3338","Type":"ContainerDied","Data":"4f8f493570aaca7af044ab4de63b20e7d4181d7b543b424235807afecc290937"} Nov 02 21:01:27.372912 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.372678 5561 scope.go:122] "RemoveContainer" containerID="ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3" Nov 02 21:01:27.372912 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.372636 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89" Nov 02 21:01:27.381012 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.380991 5561 scope.go:122] "RemoveContainer" containerID="ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3" Nov 02 21:01:27.381274 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:01:27.381253 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3\": container with ID starting with ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3 not found: ID does not exist" containerID="ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3" Nov 02 21:01:27.381320 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.381287 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3"} err="failed to get container status \"ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3\": rpc error: code = NotFound desc = could not find container \"ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3\": container with ID starting with ddfb601f0ef67557dcf06b09c69a7b77a88bb1287cb31d751a0366083e8625b3 not found: ID does not exist" Nov 02 21:01:27.526912 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.526866 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89"] Nov 02 21:01:27.573894 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:27.573838 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2c9f5f8f-kserve-668cc5c9c-bsk89"] Nov 02 21:01:29.312105 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:29.312076 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2152792-5adb-40d3-b8bf-b8c5554d3338" path="/var/lib/kubelet/pods/e2152792-5adb-40d3-b8bf-b8c5554d3338/volumes" Nov 02 21:01:33.778255 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:01:33.778216 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:01:35.265996 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.265942 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" podUID="c270f91a-06a9-4988-9afa-68fa72b41261" containerName="main" containerID="cri-o://521389a394e2bae548c927809e89502b823c6e854a1b3650e6706884fcbcb53a" gracePeriod=45 Nov 02 21:01:35.425275 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.425245 5561 generic.go:356] "Generic (PLEG): container finished" podID="c270f91a-06a9-4988-9afa-68fa72b41261" containerID="521389a394e2bae548c927809e89502b823c6e854a1b3650e6706884fcbcb53a" exitCode=0 Nov 02 21:01:35.425445 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.425322 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" event={"ID":"c270f91a-06a9-4988-9afa-68fa72b41261","Type":"ContainerDied","Data":"521389a394e2bae548c927809e89502b823c6e854a1b3650e6706884fcbcb53a"} Nov 02 21:01:35.508538 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.508513 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:01:35.537816 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.537741 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-model-cache\") pod \"c270f91a-06a9-4988-9afa-68fa72b41261\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " Nov 02 21:01:35.537816 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.537795 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-home\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-home\") pod \"c270f91a-06a9-4988-9afa-68fa72b41261\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " Nov 02 21:01:35.538001 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.537841 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c270f91a-06a9-4988-9afa-68fa72b41261-kube-api-access-dhvwm\" (UniqueName: \"kubernetes.io/projected/c270f91a-06a9-4988-9afa-68fa72b41261-kube-api-access-dhvwm\") pod \"c270f91a-06a9-4988-9afa-68fa72b41261\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " Nov 02 21:01:35.538039 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.538017 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-dshm\") pod \"c270f91a-06a9-4988-9afa-68fa72b41261\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " Nov 02 21:01:35.538073 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.538050 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-home" pod "c270f91a-06a9-4988-9afa-68fa72b41261" (UID: "c270f91a-06a9-4988-9afa-68fa72b41261"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:35.538122 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.538075 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c270f91a-06a9-4988-9afa-68fa72b41261-tls-certs\" (UniqueName: \"kubernetes.io/secret/c270f91a-06a9-4988-9afa-68fa72b41261-tls-certs\") pod \"c270f91a-06a9-4988-9afa-68fa72b41261\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " Nov 02 21:01:35.538170 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.538145 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-tmp-dir\") pod \"c270f91a-06a9-4988-9afa-68fa72b41261\" (UID: \"c270f91a-06a9-4988-9afa-68fa72b41261\") " Nov 02 21:01:35.538170 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.538073 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-model-cache" pod "c270f91a-06a9-4988-9afa-68fa72b41261" (UID: "c270f91a-06a9-4988-9afa-68fa72b41261"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:35.538414 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.538390 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-tmp-dir" pod "c270f91a-06a9-4988-9afa-68fa72b41261" (UID: "c270f91a-06a9-4988-9afa-68fa72b41261"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:35.538551 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.538453 5561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-model-cache\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:35.538551 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.538469 5561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-home\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:35.539972 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.539948 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-dshm" pod "c270f91a-06a9-4988-9afa-68fa72b41261" (UID: "c270f91a-06a9-4988-9afa-68fa72b41261"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:01:35.540063 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.540016 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c270f91a-06a9-4988-9afa-68fa72b41261-kube-api-access-dhvwm" pod "c270f91a-06a9-4988-9afa-68fa72b41261" (UID: "c270f91a-06a9-4988-9afa-68fa72b41261"). InnerVolumeSpecName "kube-api-access-dhvwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:35.540103 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.540083 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c270f91a-06a9-4988-9afa-68fa72b41261-tls-certs" pod "c270f91a-06a9-4988-9afa-68fa72b41261" (UID: "c270f91a-06a9-4988-9afa-68fa72b41261"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:01:35.639492 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.639452 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhvwm\" (UniqueName: \"kubernetes.io/projected/c270f91a-06a9-4988-9afa-68fa72b41261-kube-api-access-dhvwm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:35.639492 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.639494 5561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-dshm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:35.639711 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.639510 5561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c270f91a-06a9-4988-9afa-68fa72b41261-tls-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:35.639711 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:35.639524 5561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c270f91a-06a9-4988-9afa-68fa72b41261-tmp-dir\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:36.430977 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:36.430945 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" Nov 02 21:01:36.431404 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:36.430950 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6" event={"ID":"c270f91a-06a9-4988-9afa-68fa72b41261","Type":"ContainerDied","Data":"efd1f6006e3841844ad1b63efe59bf847f45245c66bb0485e5b45cc0cfeae55e"} Nov 02 21:01:36.431404 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:36.431078 5561 scope.go:122] "RemoveContainer" containerID="521389a394e2bae548c927809e89502b823c6e854a1b3650e6706884fcbcb53a" Nov 02 21:01:36.587130 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:36.587095 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6"] Nov 02 21:01:36.589369 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:36.589343 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2c9f5f8f-kserve-99db9cfd5-5wng6"] Nov 02 21:01:37.311767 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:01:37.311733 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c270f91a-06a9-4988-9afa-68fa72b41261" path="/var/lib/kubelet/pods/c270f91a-06a9-4988-9afa-68fa72b41261/volumes" Nov 02 21:02:03.778060 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:02:03.777982 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:02:04.494908 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:04.494855 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:25.433561 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:25.433522 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf"] Nov 02 21:02:25.434208 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:25.433715 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" podUID="5a7307c1-b1e6-4858-9f67-f9b8022f5279" containerName="maas-api" containerID="cri-o://65ea736a06a51e434d299c537d1c60564c80b0a6bd5ce15b512681571d4aebf9" gracePeriod=30 Nov 02 21:02:25.548451 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:25.548413 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:26.178982 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.178940 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:26.438794 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.438709 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:26.675424 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.675393 5561 generic.go:356] "Generic (PLEG): container finished" podID="5a7307c1-b1e6-4858-9f67-f9b8022f5279" containerID="65ea736a06a51e434d299c537d1c60564c80b0a6bd5ce15b512681571d4aebf9" exitCode=0 Nov 02 21:02:26.675580 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.675508 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" event={"ID":"5a7307c1-b1e6-4858-9f67-f9b8022f5279","Type":"ContainerDied","Data":"65ea736a06a51e434d299c537d1c60564c80b0a6bd5ce15b512681571d4aebf9"} Nov 02 21:02:26.675580 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.675569 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" event={"ID":"5a7307c1-b1e6-4858-9f67-f9b8022f5279","Type":"ContainerDied","Data":"f417351c2ee73e7520860e772a9a81066d92bc5f98cbdc3ae205802d87b3b60b"} Nov 02 21:02:26.675682 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.675589 5561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f417351c2ee73e7520860e772a9a81066d92bc5f98cbdc3ae205802d87b3b60b" Nov 02 21:02:26.678947 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.678928 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 21:02:26.724253 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.724175 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7307c1-b1e6-4858-9f67-f9b8022f5279-kube-api-access-mjmvs\" (UniqueName: \"kubernetes.io/projected/5a7307c1-b1e6-4858-9f67-f9b8022f5279-kube-api-access-mjmvs\") pod \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " Nov 02 21:02:26.724253 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.724217 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-metrics-certs\") pod \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " Nov 02 21:02:26.724253 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.724243 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7307c1-b1e6-4858-9f67-f9b8022f5279-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7307c1-b1e6-4858-9f67-f9b8022f5279-openshift-service-ca\") pod \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " Nov 02 21:02:26.724566 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.724276 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-maas-api-tls\") pod \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\" (UID: \"5a7307c1-b1e6-4858-9f67-f9b8022f5279\") " Nov 02 21:02:26.724659 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.724632 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7307c1-b1e6-4858-9f67-f9b8022f5279-openshift-service-ca" pod "5a7307c1-b1e6-4858-9f67-f9b8022f5279" (UID: "5a7307c1-b1e6-4858-9f67-f9b8022f5279"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:02:26.726438 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.726412 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-metrics-certs" pod "5a7307c1-b1e6-4858-9f67-f9b8022f5279" (UID: "5a7307c1-b1e6-4858-9f67-f9b8022f5279"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:02:26.726541 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.726456 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7307c1-b1e6-4858-9f67-f9b8022f5279-kube-api-access-mjmvs" pod "5a7307c1-b1e6-4858-9f67-f9b8022f5279" (UID: "5a7307c1-b1e6-4858-9f67-f9b8022f5279"). InnerVolumeSpecName "kube-api-access-mjmvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:02:26.726584 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.726537 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-maas-api-tls" pod "5a7307c1-b1e6-4858-9f67-f9b8022f5279" (UID: "5a7307c1-b1e6-4858-9f67-f9b8022f5279"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:02:26.825466 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.825440 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjmvs\" (UniqueName: \"kubernetes.io/projected/5a7307c1-b1e6-4858-9f67-f9b8022f5279-kube-api-access-mjmvs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:26.825466 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.825464 5561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-metrics-certs\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:26.825466 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.825474 5561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7307c1-b1e6-4858-9f67-f9b8022f5279-openshift-service-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:26.825694 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:26.825483 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7307c1-b1e6-4858-9f67-f9b8022f5279-maas-api-tls\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:27.679244 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:27.679203 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf" Nov 02 21:02:27.835152 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:27.834176 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf"] Nov 02 21:02:27.880535 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:27.880497 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-556d9884fd-cdbxf"] Nov 02 21:02:29.310929 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:29.310865 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a7307c1-b1e6-4858-9f67-f9b8022f5279" path="/var/lib/kubelet/pods/5a7307c1-b1e6-4858-9f67-f9b8022f5279/volumes" Nov 02 21:02:29.748138 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:29.748102 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:29.961055 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:29.961023 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:33.002598 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.002562 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:33.506766 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.506735 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m"] Nov 02 21:02:33.507179 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.507161 5561 cpu_manager.go:414] "containerMap: removing container" podUID="c270f91a-06a9-4988-9afa-68fa72b41261" containerName="main" Nov 02 21:02:33.507237 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.507181 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c270f91a-06a9-4988-9afa-68fa72b41261" containerName="main" Nov 02 21:02:33.507237 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.507209 5561 cpu_manager.go:414] "containerMap: removing container" podUID="5a7307c1-b1e6-4858-9f67-f9b8022f5279" containerName="maas-api" Nov 02 21:02:33.507237 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.507214 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7307c1-b1e6-4858-9f67-f9b8022f5279" containerName="maas-api" Nov 02 21:02:33.507237 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.507234 5561 cpu_manager.go:414] "containerMap: removing container" podUID="e2152792-5adb-40d3-b8bf-b8c5554d3338" containerName="main" Nov 02 21:02:33.507237 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.507239 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2152792-5adb-40d3-b8bf-b8c5554d3338" containerName="main" Nov 02 21:02:33.507433 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.507315 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2152792-5adb-40d3-b8bf-b8c5554d3338" containerName="main" Nov 02 21:02:33.507433 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.507325 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="c270f91a-06a9-4988-9afa-68fa72b41261" containerName="main" Nov 02 21:02:33.507433 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.507333 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7307c1-b1e6-4858-9f67-f9b8022f5279" containerName="maas-api" Nov 02 21:02:33.510860 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.510839 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" Nov 02 21:02:33.562926 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.562866 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3203ddac\"" Nov 02 21:02:33.586411 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.586374 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8dcg\" (UniqueName: \"kubernetes.io/projected/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-kube-api-access-f8dcg\") pod \"payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m\" (UID: \"b0a1b64f-8eb8-4431-ae73-67a31e795ba8\") " pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" Nov 02 21:02:33.586601 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.586529 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-plugins-config-volume\") pod \"payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m\" (UID: \"b0a1b64f-8eb8-4431-ae73-67a31e795ba8\") " pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" Nov 02 21:02:33.610232 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.610204 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3203ddac-dockercfg-pcjp7\"" Nov 02 21:02:33.611312 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.611288 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m"] Nov 02 21:02:33.687110 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.687079 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-plugins-config-volume\") pod \"payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m\" (UID: \"b0a1b64f-8eb8-4431-ae73-67a31e795ba8\") " pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" Nov 02 21:02:33.687292 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.687140 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8dcg\" (UniqueName: \"kubernetes.io/projected/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-kube-api-access-f8dcg\") pod \"payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m\" (UID: \"b0a1b64f-8eb8-4431-ae73-67a31e795ba8\") " pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" Nov 02 21:02:33.687840 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.687821 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-plugins-config-volume\") pod \"payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m\" (UID: \"b0a1b64f-8eb8-4431-ae73-67a31e795ba8\") " pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" Nov 02 21:02:33.741656 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.741622 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8dcg\" (UniqueName: \"kubernetes.io/projected/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-kube-api-access-f8dcg\") pod \"payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m\" (UID: \"b0a1b64f-8eb8-4431-ae73-67a31e795ba8\") " pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" Nov 02 21:02:33.778459 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:02:33.778368 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:02:33.826528 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:33.826495 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" Nov 02 21:02:34.202482 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:34.202444 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m"] Nov 02 21:02:34.210738 ip-10-0-134-194 kubenswrapper[5561]: W1102 21:02:34.210712 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a1b64f_8eb8_4431_ae73_67a31e795ba8.slice/crio-93aaaf197ef1212a4836840d1ee70e300823033d6d5ee309ab5c0c69ff2c01d3 WatchSource:0}: Error finding container 93aaaf197ef1212a4836840d1ee70e300823033d6d5ee309ab5c0c69ff2c01d3: Status 404 returned error can't find the container with id 93aaaf197ef1212a4836840d1ee70e300823033d6d5ee309ab5c0c69ff2c01d3 Nov 02 21:02:34.710141 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:34.710096 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" event={"ID":"b0a1b64f-8eb8-4431-ae73-67a31e795ba8","Type":"ContainerStarted","Data":"9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335"} Nov 02 21:02:34.710141 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:34.710148 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" event={"ID":"b0a1b64f-8eb8-4431-ae73-67a31e795ba8","Type":"ContainerStarted","Data":"93aaaf197ef1212a4836840d1ee70e300823033d6d5ee309ab5c0c69ff2c01d3"} Nov 02 21:02:34.710395 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:34.710195 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" Nov 02 21:02:34.835707 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:34.835655 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" podStartSLOduration=1.835641058 podStartE2EDuration="1.835641058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:02:34.817572268 +0000 UTC m=+4191.625356512" watchObservedRunningTime="2026-11-02 21:02:34.835641058 +0000 UTC m=+4191.643425313" Nov 02 21:02:35.065207 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:35.065121 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:37.851531 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:37.851499 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:42.482147 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:42.482099 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:44.531421 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:44.531387 5561 scope.go:122] "RemoveContainer" containerID="65ea736a06a51e434d299c537d1c60564c80b0a6bd5ce15b512681571d4aebf9" Nov 02 21:02:45.717798 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:45.717771 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" Nov 02 21:02:47.026838 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:47.026801 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:52.008157 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:52.008120 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:56.376358 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:56.376323 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:56.548161 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:02:56.548126 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:01.251287 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:01.251236 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:03.779310 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:03:03.779275 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:03:04.823471 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:04.823434 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:05.311243 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:05.311204 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:06.095293 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:06.095262 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:10.652920 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:10.652858 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:11.052789 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.052711 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl"] Nov 02 21:03:11.056292 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.056275 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:11.107138 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.107107 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:03:11.107138 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.107128 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8g4zg\"" Nov 02 21:03:11.110845 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.110811 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl\" (UID: \"dd9255fe-4239-400a-8ce2-7cb83e225993\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:11.111001 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.110890 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92t9l\" (UniqueName: \"kubernetes.io/projected/dd9255fe-4239-400a-8ce2-7cb83e225993-kube-api-access-92t9l\") pod \"maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl\" (UID: \"dd9255fe-4239-400a-8ce2-7cb83e225993\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:11.111001 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.110962 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl\" (UID: \"dd9255fe-4239-400a-8ce2-7cb83e225993\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:11.154655 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.154615 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:03:11.154655 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.154625 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:03:11.155751 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.155730 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl"] Nov 02 21:03:11.211959 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.211920 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92t9l\" (UniqueName: \"kubernetes.io/projected/dd9255fe-4239-400a-8ce2-7cb83e225993-kube-api-access-92t9l\") pod \"maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl\" (UID: \"dd9255fe-4239-400a-8ce2-7cb83e225993\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:11.212169 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.211966 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl\" (UID: \"dd9255fe-4239-400a-8ce2-7cb83e225993\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:11.212169 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.212056 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl\" (UID: \"dd9255fe-4239-400a-8ce2-7cb83e225993\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:11.212669 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.212649 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl\" (UID: \"dd9255fe-4239-400a-8ce2-7cb83e225993\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:11.214389 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.214371 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl\" (UID: \"dd9255fe-4239-400a-8ce2-7cb83e225993\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:11.265751 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.265714 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92t9l\" (UniqueName: \"kubernetes.io/projected/dd9255fe-4239-400a-8ce2-7cb83e225993-kube-api-access-92t9l\") pod \"maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl\" (UID: \"dd9255fe-4239-400a-8ce2-7cb83e225993\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:11.368697 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.368595 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:11.736099 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.736061 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl"] Nov 02 21:03:11.743943 ip-10-0-134-194 kubenswrapper[5561]: W1102 21:03:11.743911 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9255fe_4239_400a_8ce2_7cb83e225993.slice/crio-6e037c6613003432feed6677601d65d1d6441034a35a978e490a1a2d2421ec5e WatchSource:0}: Error finding container 6e037c6613003432feed6677601d65d1d6441034a35a978e490a1a2d2421ec5e: Status 404 returned error can't find the container with id 6e037c6613003432feed6677601d65d1d6441034a35a978e490a1a2d2421ec5e Nov 02 21:03:11.869439 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:11.869405 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" event={"ID":"dd9255fe-4239-400a-8ce2-7cb83e225993","Type":"ContainerStarted","Data":"6e037c6613003432feed6677601d65d1d6441034a35a978e490a1a2d2421ec5e"} Nov 02 21:03:12.875710 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:12.875618 5561 generic.go:356] "Generic (PLEG): container finished" podID="dd9255fe-4239-400a-8ce2-7cb83e225993" containerID="a0dd0a31ba8c416c397517864976e1c3b3ff9dd5228d9f00401ce62537c46450" exitCode=0 Nov 02 21:03:12.876108 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:12.875700 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" event={"ID":"dd9255fe-4239-400a-8ce2-7cb83e225993","Type":"ContainerDied","Data":"a0dd0a31ba8c416c397517864976e1c3b3ff9dd5228d9f00401ce62537c46450"} Nov 02 21:03:14.007522 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.007495 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:14.038006 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.037980 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-openshift-ca\") pod \"dd9255fe-4239-400a-8ce2-7cb83e225993\" (UID: \"dd9255fe-4239-400a-8ce2-7cb83e225993\") " Nov 02 21:03:14.038153 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.038048 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9255fe-4239-400a-8ce2-7cb83e225993-kube-api-access-92t9l\" (UniqueName: \"kubernetes.io/projected/dd9255fe-4239-400a-8ce2-7cb83e225993-kube-api-access-92t9l\") pod \"dd9255fe-4239-400a-8ce2-7cb83e225993\" (UID: \"dd9255fe-4239-400a-8ce2-7cb83e225993\") " Nov 02 21:03:14.038275 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.038162 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-xks-ca\") pod \"dd9255fe-4239-400a-8ce2-7cb83e225993\" (UID: \"dd9255fe-4239-400a-8ce2-7cb83e225993\") " Nov 02 21:03:14.038373 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.038349 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-openshift-ca" pod "dd9255fe-4239-400a-8ce2-7cb83e225993" (UID: "dd9255fe-4239-400a-8ce2-7cb83e225993"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:03:14.038447 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.038433 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-openshift-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:14.040068 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.040045 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9255fe-4239-400a-8ce2-7cb83e225993-kube-api-access-92t9l" pod "dd9255fe-4239-400a-8ce2-7cb83e225993" (UID: "dd9255fe-4239-400a-8ce2-7cb83e225993"). InnerVolumeSpecName "kube-api-access-92t9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:03:14.040155 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.040099 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-xks-ca" pod "dd9255fe-4239-400a-8ce2-7cb83e225993" (UID: "dd9255fe-4239-400a-8ce2-7cb83e225993"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:03:14.139374 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.139300 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9255fe-4239-400a-8ce2-7cb83e225993-maas-api-xks-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:14.139374 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.139328 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92t9l\" (UniqueName: \"kubernetes.io/projected/dd9255fe-4239-400a-8ce2-7cb83e225993-kube-api-access-92t9l\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:14.884922 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.884891 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" Nov 02 21:03:14.885129 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.884891 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl" event={"ID":"dd9255fe-4239-400a-8ce2-7cb83e225993","Type":"ContainerDied","Data":"6e037c6613003432feed6677601d65d1d6441034a35a978e490a1a2d2421ec5e"} Nov 02 21:03:14.885129 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:14.885014 5561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e037c6613003432feed6677601d65d1d6441034a35a978e490a1a2d2421ec5e" Nov 02 21:03:15.147706 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:15.147614 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:19.685342 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:19.685308 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:22.563714 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:22.563670 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m"] Nov 02 21:03:22.979450 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:22.979416 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:23.159438 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:23.159401 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:24.161050 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:24.160970 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:27.564540 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.564478 5561 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" podUID="b0a1b64f-8eb8-4431-ae73-67a31e795ba8" containerName="payload-processing" containerID="cri-o://9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335" gracePeriod=25 Nov 02 21:03:27.813702 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.813674 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" Nov 02 21:03:27.861828 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.861745 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-kube-api-access-f8dcg\" (UniqueName: \"kubernetes.io/projected/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-kube-api-access-f8dcg\") pod \"b0a1b64f-8eb8-4431-ae73-67a31e795ba8\" (UID: \"b0a1b64f-8eb8-4431-ae73-67a31e795ba8\") " Nov 02 21:03:27.861828 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.861795 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-plugins-config-volume\") pod \"b0a1b64f-8eb8-4431-ae73-67a31e795ba8\" (UID: \"b0a1b64f-8eb8-4431-ae73-67a31e795ba8\") " Nov 02 21:03:27.862213 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.862192 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-plugins-config-volume" pod "b0a1b64f-8eb8-4431-ae73-67a31e795ba8" (UID: "b0a1b64f-8eb8-4431-ae73-67a31e795ba8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:03:27.863864 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.863839 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-kube-api-access-f8dcg" pod "b0a1b64f-8eb8-4431-ae73-67a31e795ba8" (UID: "b0a1b64f-8eb8-4431-ae73-67a31e795ba8"). InnerVolumeSpecName "kube-api-access-f8dcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:03:27.941308 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.941276 5561 generic.go:356] "Generic (PLEG): container finished" podID="b0a1b64f-8eb8-4431-ae73-67a31e795ba8" containerID="9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335" exitCode=0 Nov 02 21:03:27.941480 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.941319 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" Nov 02 21:03:27.941480 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.941317 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" event={"ID":"b0a1b64f-8eb8-4431-ae73-67a31e795ba8","Type":"ContainerDied","Data":"9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335"} Nov 02 21:03:27.941480 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.941421 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m" event={"ID":"b0a1b64f-8eb8-4431-ae73-67a31e795ba8","Type":"ContainerDied","Data":"93aaaf197ef1212a4836840d1ee70e300823033d6d5ee309ab5c0c69ff2c01d3"} Nov 02 21:03:27.941480 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.941456 5561 scope.go:122] "RemoveContainer" containerID="9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335" Nov 02 21:03:27.950446 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.950428 5561 scope.go:122] "RemoveContainer" containerID="9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335" Nov 02 21:03:27.950721 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:03:27.950699 5561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335\": container with ID starting with 9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335 not found: ID does not exist" containerID="9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335" Nov 02 21:03:27.950779 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.950730 5561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335"} err="failed to get container status \"9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335\": rpc error: code = NotFound desc = could not find container \"9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335\": container with ID starting with 9cb04edac83d9c30708879cfb87d4faa80bac070d62e5810979c01002b97a335 not found: ID does not exist" Nov 02 21:03:27.962721 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.962692 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8dcg\" (UniqueName: \"kubernetes.io/projected/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-kube-api-access-f8dcg\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:27.962721 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:27.962721 5561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0a1b64f-8eb8-4431-ae73-67a31e795ba8-plugins-config-volume\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:28.095936 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:28.095894 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m"] Nov 02 21:03:28.101434 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:28.101401 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3203ddac-667f87d4ff-pcj6m"] Nov 02 21:03:28.120110 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:28.120042 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:29.311163 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:29.311124 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a1b64f-8eb8-4431-ae73-67a31e795ba8" path="/var/lib/kubelet/pods/b0a1b64f-8eb8-4431-ae73-67a31e795ba8/volumes" Nov 02 21:03:29.553544 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:29.553510 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:29.810504 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:29.810439 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:29.969083 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:29.969046 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl"] Nov 02 21:03:30.016601 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:30.016560 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3203ddac-pjtxl"] Nov 02 21:03:31.312455 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:31.312420 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9255fe-4239-400a-8ce2-7cb83e225993" path="/var/lib/kubelet/pods/dd9255fe-4239-400a-8ce2-7cb83e225993/volumes" Nov 02 21:03:32.179174 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:32.179133 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:33.434801 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:33.434768 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:33.780678 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:03:33.780546 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:03:34.024454 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:34.024421 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:34.274561 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:34.274524 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:37.650962 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:37.650927 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:37.835758 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:37.835718 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:40.812077 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:40.812041 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:40.988888 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:40.988852 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:43.623605 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:43.623565 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:43.805128 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:43.805096 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:43.970333 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:43.970283 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:48.269736 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:48.269682 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:52.987081 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:52.987042 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:57.580973 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:57.580932 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:59.421524 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:03:59.421484 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:00.100510 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:00.100481 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:02.467418 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:02.467370 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:02.648747 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:02.648705 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:03.779383 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:04:03.779350 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:04:05.154017 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:05.153979 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:05.649906 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:05.649799 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:06.998108 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:06.998076 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:11.576705 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:11.576666 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:16.050755 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:16.050720 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:20.598557 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:20.598517 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:25.075970 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:25.075922 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:29.641005 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:29.640828 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:33.778964 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:04:33.778862 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:04:34.158346 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:34.158310 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:38.754002 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:38.753951 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:43.280963 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:43.280928 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:47.846935 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:47.846894 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:52.374413 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:52.374377 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:56.946635 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:04:56.946543 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:01.473833 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:01.473786 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:03.779718 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:05:03.779679 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:05:04.533004 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:04.532970 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:05.495703 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:05.495667 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:06.110967 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:06.110932 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:10.577476 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:10.577435 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:15.159660 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:15.159621 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:19.662410 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:19.662370 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:23.281933 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.281895 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k"] Nov 02 21:05:23.282405 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.282390 5561 cpu_manager.go:414] "containerMap: removing container" podUID="dd9255fe-4239-400a-8ce2-7cb83e225993" containerName="revoke-keys" Nov 02 21:05:23.282450 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.282408 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9255fe-4239-400a-8ce2-7cb83e225993" containerName="revoke-keys" Nov 02 21:05:23.282450 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.282431 5561 cpu_manager.go:414] "containerMap: removing container" podUID="b0a1b64f-8eb8-4431-ae73-67a31e795ba8" containerName="payload-processing" Nov 02 21:05:23.282450 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.282439 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a1b64f-8eb8-4431-ae73-67a31e795ba8" containerName="payload-processing" Nov 02 21:05:23.282549 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.282532 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9255fe-4239-400a-8ce2-7cb83e225993" containerName="revoke-keys" Nov 02 21:05:23.282549 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.282547 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a1b64f-8eb8-4431-ae73-67a31e795ba8" containerName="payload-processing" Nov 02 21:05:23.285651 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.285632 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:23.338175 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.338142 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k"] Nov 02 21:05:23.384466 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.384438 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:05:23.384669 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.384551 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:05:23.384669 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.384602 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:05:23.384669 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.384618 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8g4zg\"" Nov 02 21:05:23.405907 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.405861 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-496nm\" (UniqueName: \"kubernetes.io/projected/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-kube-api-access-496nm\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k\" (UID: \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:23.406069 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.405918 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k\" (UID: \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:23.406069 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.405998 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k\" (UID: \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:23.506588 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.506558 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-496nm\" (UniqueName: \"kubernetes.io/projected/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-kube-api-access-496nm\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k\" (UID: \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:23.506795 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.506595 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k\" (UID: \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:23.506795 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.506614 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k\" (UID: \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:23.507284 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.507259 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k\" (UID: \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:23.509020 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.508994 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k\" (UID: \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:23.560385 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.560323 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-496nm\" (UniqueName: \"kubernetes.io/projected/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-kube-api-access-496nm\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k\" (UID: \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:23.598173 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.598144 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:23.966315 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:23.966281 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k"] Nov 02 21:05:23.973970 ip-10-0-134-194 kubenswrapper[5561]: W1102 21:05:23.973943 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9d132a_dbc9_4471_9e11_5b43a1e3db7a.slice/crio-034a7366d04d8b1a501b0320c152d1d3192452a56a36e10b230e453bbbf02b86 WatchSource:0}: Error finding container 034a7366d04d8b1a501b0320c152d1d3192452a56a36e10b230e453bbbf02b86: Status 404 returned error can't find the container with id 034a7366d04d8b1a501b0320c152d1d3192452a56a36e10b230e453bbbf02b86 Nov 02 21:05:24.344370 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:24.344274 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:24.443139 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:24.443103 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" event={"ID":"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a","Type":"ContainerStarted","Data":"3de80b8d5246ca8a81c983c5495087e2d85305b47f18fba53c6f862ea7ac4724"} Nov 02 21:05:24.443312 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:24.443165 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" event={"ID":"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a","Type":"ContainerStarted","Data":"034a7366d04d8b1a501b0320c152d1d3192452a56a36e10b230e453bbbf02b86"} Nov 02 21:05:24.550846 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:24.550795 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" podStartSLOduration=1.55077831 podStartE2EDuration="1.55077831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:05:24.549804611 +0000 UTC m=+4361.357588827" watchObservedRunningTime="2026-11-02 21:05:24.55077831 +0000 UTC m=+4361.358562549" Nov 02 21:05:25.448539 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:25.448501 5561 generic.go:356] "Generic (PLEG): container finished" podID="7a9d132a-dbc9-4471-9e11-5b43a1e3db7a" containerID="3de80b8d5246ca8a81c983c5495087e2d85305b47f18fba53c6f862ea7ac4724" exitCode=0 Nov 02 21:05:25.448986 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:25.448583 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" event={"ID":"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a","Type":"ContainerDied","Data":"3de80b8d5246ca8a81c983c5495087e2d85305b47f18fba53c6f862ea7ac4724"} Nov 02 21:05:26.581012 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:26.580988 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:26.739270 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:26.739191 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-xks-ca\") pod \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\" (UID: \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\") " Nov 02 21:05:26.739270 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:26.739255 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-openshift-ca\") pod \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\" (UID: \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\") " Nov 02 21:05:26.739456 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:26.739292 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-kube-api-access-496nm\" (UniqueName: \"kubernetes.io/projected/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-kube-api-access-496nm\") pod \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\" (UID: \"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a\") " Nov 02 21:05:26.739656 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:26.739623 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-openshift-ca" pod "7a9d132a-dbc9-4471-9e11-5b43a1e3db7a" (UID: "7a9d132a-dbc9-4471-9e11-5b43a1e3db7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:05:26.741395 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:26.741372 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-kube-api-access-496nm" pod "7a9d132a-dbc9-4471-9e11-5b43a1e3db7a" (UID: "7a9d132a-dbc9-4471-9e11-5b43a1e3db7a"). InnerVolumeSpecName "kube-api-access-496nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:05:26.741485 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:26.741467 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-xks-ca" pod "7a9d132a-dbc9-4471-9e11-5b43a1e3db7a" (UID: "7a9d132a-dbc9-4471-9e11-5b43a1e3db7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:05:26.840162 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:26.840126 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-xks-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:05:26.840162 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:26.840158 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-maas-api-openshift-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:05:26.840375 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:26.840172 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-496nm\" (UniqueName: \"kubernetes.io/projected/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a-kube-api-access-496nm\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:05:27.457952 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:27.457918 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" event={"ID":"7a9d132a-dbc9-4471-9e11-5b43a1e3db7a","Type":"ContainerDied","Data":"034a7366d04d8b1a501b0320c152d1d3192452a56a36e10b230e453bbbf02b86"} Nov 02 21:05:27.457952 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:27.457941 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k" Nov 02 21:05:27.458164 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:27.457971 5561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="034a7366d04d8b1a501b0320c152d1d3192452a56a36e10b230e453bbbf02b86" Nov 02 21:05:29.255069 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:29.255033 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:33.776245 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:33.776205 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:33.779345 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:05:33.779311 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:05:35.264320 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:35.264281 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:35.434982 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:35.434946 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:38.261778 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:38.261724 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:43.086977 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:43.086931 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:47.291361 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:47.291325 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:48.722772 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:48.722733 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:48.989893 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:48.989767 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:49.241575 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:49.241494 5561 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k"] Nov 02 21:05:49.288569 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:49.288534 5561 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ac1065-zd58k"] Nov 02 21:05:49.311834 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:49.311798 5561 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a9d132a-dbc9-4471-9e11-5b43a1e3db7a" path="/var/lib/kubelet/pods/7a9d132a-dbc9-4471-9e11-5b43a1e3db7a/volumes" Nov 02 21:05:49.729770 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:49.729740 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:55.564679 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:55.564635 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:55.740501 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:55.740464 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:55.926724 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:05:55.926686 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:06:03.778327 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:06:03.778291 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:06:05.848802 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:05.848759 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:06:06.319100 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:06.319069 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:06:15.956777 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:15.956741 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:06:29.824028 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.823948 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd"] Nov 02 21:06:29.824399 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.824300 5561 cpu_manager.go:414] "containerMap: removing container" podUID="7a9d132a-dbc9-4471-9e11-5b43a1e3db7a" containerName="revoke-keys" Nov 02 21:06:29.824399 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.824311 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9d132a-dbc9-4471-9e11-5b43a1e3db7a" containerName="revoke-keys" Nov 02 21:06:29.824399 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.824368 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9d132a-dbc9-4471-9e11-5b43a1e3db7a" containerName="revoke-keys" Nov 02 21:06:29.827280 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.827263 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:29.927351 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.927295 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:06:29.927545 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.927344 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:06:29.927545 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.927355 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:06:29.927545 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.927394 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8g4zg\"" Nov 02 21:06:29.928478 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.928457 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd"] Nov 02 21:06:29.949480 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.949445 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xslhd\" (UID: \"b97d9a99-6503-4245-9c69-8cfb2e673455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:29.949639 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.949502 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xslhd\" (UID: \"b97d9a99-6503-4245-9c69-8cfb2e673455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:29.949639 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:29.949548 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24wjf\" (UniqueName: \"kubernetes.io/projected/b97d9a99-6503-4245-9c69-8cfb2e673455-kube-api-access-24wjf\") pod \"maas-api-revoke-sub-f03629db-xslhd\" (UID: \"b97d9a99-6503-4245-9c69-8cfb2e673455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:30.050353 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:30.050305 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xslhd\" (UID: \"b97d9a99-6503-4245-9c69-8cfb2e673455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:30.050569 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:30.050378 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xslhd\" (UID: \"b97d9a99-6503-4245-9c69-8cfb2e673455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:30.050569 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:30.050403 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24wjf\" (UniqueName: \"kubernetes.io/projected/b97d9a99-6503-4245-9c69-8cfb2e673455-kube-api-access-24wjf\") pod \"maas-api-revoke-sub-f03629db-xslhd\" (UID: \"b97d9a99-6503-4245-9c69-8cfb2e673455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:30.051118 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:30.051096 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xslhd\" (UID: \"b97d9a99-6503-4245-9c69-8cfb2e673455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:30.052784 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:30.052764 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xslhd\" (UID: \"b97d9a99-6503-4245-9c69-8cfb2e673455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:30.104915 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:30.104832 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24wjf\" (UniqueName: \"kubernetes.io/projected/b97d9a99-6503-4245-9c69-8cfb2e673455-kube-api-access-24wjf\") pod \"maas-api-revoke-sub-f03629db-xslhd\" (UID: \"b97d9a99-6503-4245-9c69-8cfb2e673455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:30.139668 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:30.139632 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:30.507515 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:30.507474 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd"] Nov 02 21:06:30.516175 ip-10-0-134-194 kubenswrapper[5561]: W1102 21:06:30.516142 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97d9a99_6503_4245_9c69_8cfb2e673455.slice/crio-c1e9fc153703f2fea404c3df9ac6f07e61180e56b7754ca2dddbe52d6353cd4f WatchSource:0}: Error finding container c1e9fc153703f2fea404c3df9ac6f07e61180e56b7754ca2dddbe52d6353cd4f: Status 404 returned error can't find the container with id c1e9fc153703f2fea404c3df9ac6f07e61180e56b7754ca2dddbe52d6353cd4f Nov 02 21:06:30.775363 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:30.775277 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" event={"ID":"b97d9a99-6503-4245-9c69-8cfb2e673455","Type":"ContainerStarted","Data":"c1e9fc153703f2fea404c3df9ac6f07e61180e56b7754ca2dddbe52d6353cd4f"} Nov 02 21:06:31.781718 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:31.781620 5561 generic.go:356] "Generic (PLEG): container finished" podID="b97d9a99-6503-4245-9c69-8cfb2e673455" containerID="c26ebd37a490ac71de15b043f129d11b67d42052c243fad744028394d676fcb4" exitCode=0 Nov 02 21:06:31.782152 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:31.781706 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" event={"ID":"b97d9a99-6503-4245-9c69-8cfb2e673455","Type":"ContainerDied","Data":"c26ebd37a490ac71de15b043f129d11b67d42052c243fad744028394d676fcb4"} Nov 02 21:06:32.913059 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:32.913031 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:32.972236 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:32.972203 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-xks-ca\") pod \"b97d9a99-6503-4245-9c69-8cfb2e673455\" (UID: \"b97d9a99-6503-4245-9c69-8cfb2e673455\") " Nov 02 21:06:32.974266 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:32.974230 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-xks-ca" pod "b97d9a99-6503-4245-9c69-8cfb2e673455" (UID: "b97d9a99-6503-4245-9c69-8cfb2e673455"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:06:33.072847 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:33.072768 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-openshift-ca\") pod \"b97d9a99-6503-4245-9c69-8cfb2e673455\" (UID: \"b97d9a99-6503-4245-9c69-8cfb2e673455\") " Nov 02 21:06:33.072847 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:33.072819 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97d9a99-6503-4245-9c69-8cfb2e673455-kube-api-access-24wjf\" (UniqueName: \"kubernetes.io/projected/b97d9a99-6503-4245-9c69-8cfb2e673455-kube-api-access-24wjf\") pod \"b97d9a99-6503-4245-9c69-8cfb2e673455\" (UID: \"b97d9a99-6503-4245-9c69-8cfb2e673455\") " Nov 02 21:06:33.073086 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:33.073073 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-xks-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:33.073161 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:33.073138 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-openshift-ca" pod "b97d9a99-6503-4245-9c69-8cfb2e673455" (UID: "b97d9a99-6503-4245-9c69-8cfb2e673455"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:06:33.074860 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:33.074839 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97d9a99-6503-4245-9c69-8cfb2e673455-kube-api-access-24wjf" pod "b97d9a99-6503-4245-9c69-8cfb2e673455" (UID: "b97d9a99-6503-4245-9c69-8cfb2e673455"). InnerVolumeSpecName "kube-api-access-24wjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:06:33.173642 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:33.173596 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97d9a99-6503-4245-9c69-8cfb2e673455-maas-api-openshift-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:33.173642 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:33.173633 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24wjf\" (UniqueName: \"kubernetes.io/projected/b97d9a99-6503-4245-9c69-8cfb2e673455-kube-api-access-24wjf\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:33.778753 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:06:33.778710 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:06:33.792748 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:33.792717 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" event={"ID":"b97d9a99-6503-4245-9c69-8cfb2e673455","Type":"ContainerDied","Data":"c1e9fc153703f2fea404c3df9ac6f07e61180e56b7754ca2dddbe52d6353cd4f"} Nov 02 21:06:33.792938 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:33.792750 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xslhd" Nov 02 21:06:33.792938 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:33.792764 5561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1e9fc153703f2fea404c3df9ac6f07e61180e56b7754ca2dddbe52d6353cd4f" Nov 02 21:06:38.788530 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:38.788498 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:06:50.327336 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.327303 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb"] Nov 02 21:06:50.327719 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.327632 5561 cpu_manager.go:414] "containerMap: removing container" podUID="b97d9a99-6503-4245-9c69-8cfb2e673455" containerName="revoke-keys" Nov 02 21:06:50.327719 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.327643 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97d9a99-6503-4245-9c69-8cfb2e673455" containerName="revoke-keys" Nov 02 21:06:50.327719 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.327714 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97d9a99-6503-4245-9c69-8cfb2e673455" containerName="revoke-keys" Nov 02 21:06:50.331039 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.331019 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:50.382885 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.382832 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb"] Nov 02 21:06:50.423535 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.423503 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-z92jb\" (UID: \"5b539820-44df-48cc-b627-fd10e38c7bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:50.423713 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.423556 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v688f\" (UniqueName: \"kubernetes.io/projected/5b539820-44df-48cc-b627-fd10e38c7bb4-kube-api-access-v688f\") pod \"maas-api-revoke-sub-0165d7d2-z92jb\" (UID: \"5b539820-44df-48cc-b627-fd10e38c7bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:50.423713 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.423681 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-z92jb\" (UID: \"5b539820-44df-48cc-b627-fd10e38c7bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:50.428581 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.428548 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:06:50.428581 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.428577 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:06:50.428754 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.428619 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:06:50.428754 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.428670 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8g4zg\"" Nov 02 21:06:50.525038 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.525000 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v688f\" (UniqueName: \"kubernetes.io/projected/5b539820-44df-48cc-b627-fd10e38c7bb4-kube-api-access-v688f\") pod \"maas-api-revoke-sub-0165d7d2-z92jb\" (UID: \"5b539820-44df-48cc-b627-fd10e38c7bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:50.525228 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.525092 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-z92jb\" (UID: \"5b539820-44df-48cc-b627-fd10e38c7bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:50.525228 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.525146 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-z92jb\" (UID: \"5b539820-44df-48cc-b627-fd10e38c7bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:50.525737 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.525716 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-z92jb\" (UID: \"5b539820-44df-48cc-b627-fd10e38c7bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:50.527362 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.527339 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-z92jb\" (UID: \"5b539820-44df-48cc-b627-fd10e38c7bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:50.579681 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.579617 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v688f\" (UniqueName: \"kubernetes.io/projected/5b539820-44df-48cc-b627-fd10e38c7bb4-kube-api-access-v688f\") pod \"maas-api-revoke-sub-0165d7d2-z92jb\" (UID: \"5b539820-44df-48cc-b627-fd10e38c7bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:50.644226 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:50.644191 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:51.013196 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:51.013159 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb"] Nov 02 21:06:51.021088 ip-10-0-134-194 kubenswrapper[5561]: W1102 21:06:51.021055 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b539820_44df_48cc_b627_fd10e38c7bb4.slice/crio-4b995eef0729dbdbfc36288686f44f6d99ea54254911af38165df08314b4e159 WatchSource:0}: Error finding container 4b995eef0729dbdbfc36288686f44f6d99ea54254911af38165df08314b4e159: Status 404 returned error can't find the container with id 4b995eef0729dbdbfc36288686f44f6d99ea54254911af38165df08314b4e159 Nov 02 21:06:51.894491 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:51.894455 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" event={"ID":"5b539820-44df-48cc-b627-fd10e38c7bb4","Type":"ContainerStarted","Data":"212bf294a1ec30e75036b83ba41f00b2d70b9d4a700881a149cdbad419ba24e2"} Nov 02 21:06:51.894889 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:51.894504 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" event={"ID":"5b539820-44df-48cc-b627-fd10e38c7bb4","Type":"ContainerStarted","Data":"4b995eef0729dbdbfc36288686f44f6d99ea54254911af38165df08314b4e159"} Nov 02 21:06:52.003494 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:52.003437 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" podStartSLOduration=2.003424581 podStartE2EDuration="2.003424581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:06:52.002019851 +0000 UTC m=+4448.809804068" watchObservedRunningTime="2026-11-02 21:06:52.003424581 +0000 UTC m=+4448.811208788" Nov 02 21:06:52.900477 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:52.900442 5561 generic.go:356] "Generic (PLEG): container finished" podID="5b539820-44df-48cc-b627-fd10e38c7bb4" containerID="212bf294a1ec30e75036b83ba41f00b2d70b9d4a700881a149cdbad419ba24e2" exitCode=0 Nov 02 21:06:52.900913 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:52.900533 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" event={"ID":"5b539820-44df-48cc-b627-fd10e38c7bb4","Type":"ContainerDied","Data":"212bf294a1ec30e75036b83ba41f00b2d70b9d4a700881a149cdbad419ba24e2"} Nov 02 21:06:54.033713 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.033687 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:54.056899 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.056854 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-xks-ca\") pod \"5b539820-44df-48cc-b627-fd10e38c7bb4\" (UID: \"5b539820-44df-48cc-b627-fd10e38c7bb4\") " Nov 02 21:06:54.056899 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.056901 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-openshift-ca\") pod \"5b539820-44df-48cc-b627-fd10e38c7bb4\" (UID: \"5b539820-44df-48cc-b627-fd10e38c7bb4\") " Nov 02 21:06:54.057130 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.056953 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b539820-44df-48cc-b627-fd10e38c7bb4-kube-api-access-v688f\" (UniqueName: \"kubernetes.io/projected/5b539820-44df-48cc-b627-fd10e38c7bb4-kube-api-access-v688f\") pod \"5b539820-44df-48cc-b627-fd10e38c7bb4\" (UID: \"5b539820-44df-48cc-b627-fd10e38c7bb4\") " Nov 02 21:06:54.057310 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.057285 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-openshift-ca" pod "5b539820-44df-48cc-b627-fd10e38c7bb4" (UID: "5b539820-44df-48cc-b627-fd10e38c7bb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:06:54.059085 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.059062 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-xks-ca" pod "5b539820-44df-48cc-b627-fd10e38c7bb4" (UID: "5b539820-44df-48cc-b627-fd10e38c7bb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:06:54.059183 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.059161 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b539820-44df-48cc-b627-fd10e38c7bb4-kube-api-access-v688f" pod "5b539820-44df-48cc-b627-fd10e38c7bb4" (UID: "5b539820-44df-48cc-b627-fd10e38c7bb4"). InnerVolumeSpecName "kube-api-access-v688f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:06:54.157853 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.157767 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-xks-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:54.157853 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.157799 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b539820-44df-48cc-b627-fd10e38c7bb4-maas-api-openshift-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:54.157853 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.157818 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v688f\" (UniqueName: \"kubernetes.io/projected/5b539820-44df-48cc-b627-fd10e38c7bb4-kube-api-access-v688f\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:54.910686 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.910649 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" Nov 02 21:06:54.910869 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.910659 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z92jb" event={"ID":"5b539820-44df-48cc-b627-fd10e38c7bb4","Type":"ContainerDied","Data":"4b995eef0729dbdbfc36288686f44f6d99ea54254911af38165df08314b4e159"} Nov 02 21:06:54.910869 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:06:54.910778 5561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b995eef0729dbdbfc36288686f44f6d99ea54254911af38165df08314b4e159" Nov 02 21:07:01.935648 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:01.935611 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp"] Nov 02 21:07:01.936074 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:01.935979 5561 cpu_manager.go:414] "containerMap: removing container" podUID="5b539820-44df-48cc-b627-fd10e38c7bb4" containerName="revoke-keys" Nov 02 21:07:01.936074 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:01.935993 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b539820-44df-48cc-b627-fd10e38c7bb4" containerName="revoke-keys" Nov 02 21:07:01.936074 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:01.936049 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b539820-44df-48cc-b627-fd10e38c7bb4" containerName="revoke-keys" Nov 02 21:07:01.939286 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:01.939264 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:02.025481 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.025435 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ssxc\" (UniqueName: \"kubernetes.io/projected/4e257c7d-2ee4-4ba0-ad07-0af088833e13-kube-api-access-4ssxc\") pod \"maas-api-revoke-sub-6b19338f-zz5lp\" (UID: \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:02.025674 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.025501 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zz5lp\" (UID: \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:02.025674 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.025587 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zz5lp\" (UID: \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:02.037220 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.037179 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:07:02.037220 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.037184 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:07:02.037402 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.037223 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:07:02.037402 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.037257 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8g4zg\"" Nov 02 21:07:02.038515 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.038494 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp"] Nov 02 21:07:02.127071 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.127034 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ssxc\" (UniqueName: \"kubernetes.io/projected/4e257c7d-2ee4-4ba0-ad07-0af088833e13-kube-api-access-4ssxc\") pod \"maas-api-revoke-sub-6b19338f-zz5lp\" (UID: \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:02.127262 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.127086 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zz5lp\" (UID: \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:02.127262 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.127203 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zz5lp\" (UID: \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:02.127702 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.127677 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zz5lp\" (UID: \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:02.129557 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.129537 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zz5lp\" (UID: \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:02.181329 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.181289 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ssxc\" (UniqueName: \"kubernetes.io/projected/4e257c7d-2ee4-4ba0-ad07-0af088833e13-kube-api-access-4ssxc\") pod \"maas-api-revoke-sub-6b19338f-zz5lp\" (UID: \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:02.252261 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.252163 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:02.621074 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.620994 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp"] Nov 02 21:07:02.628921 ip-10-0-134-194 kubenswrapper[5561]: W1102 21:07:02.628845 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e257c7d_2ee4_4ba0_ad07_0af088833e13.slice/crio-6fed258ee1755d5be2c79b29479d11f1114d0b7e11c0e0196328194ea92d29f0 WatchSource:0}: Error finding container 6fed258ee1755d5be2c79b29479d11f1114d0b7e11c0e0196328194ea92d29f0: Status 404 returned error can't find the container with id 6fed258ee1755d5be2c79b29479d11f1114d0b7e11c0e0196328194ea92d29f0 Nov 02 21:07:02.947277 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.947237 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" event={"ID":"4e257c7d-2ee4-4ba0-ad07-0af088833e13","Type":"ContainerStarted","Data":"2913e46fc659c56dc57913ef5b22d3ee8af4be2c336912fd3d74bd00039c8fb6"} Nov 02 21:07:02.947604 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:02.947294 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" event={"ID":"4e257c7d-2ee4-4ba0-ad07-0af088833e13","Type":"ContainerStarted","Data":"6fed258ee1755d5be2c79b29479d11f1114d0b7e11c0e0196328194ea92d29f0"} Nov 02 21:07:03.055479 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:03.055428 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" podStartSLOduration=2.055413545 podStartE2EDuration="2.055413545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:03.054044504 +0000 UTC m=+4459.861828742" watchObservedRunningTime="2026-11-02 21:07:03.055413545 +0000 UTC m=+4459.863197765" Nov 02 21:07:03.779497 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:07:03.779456 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:07:03.953287 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:03.953253 5561 generic.go:356] "Generic (PLEG): container finished" podID="4e257c7d-2ee4-4ba0-ad07-0af088833e13" containerID="2913e46fc659c56dc57913ef5b22d3ee8af4be2c336912fd3d74bd00039c8fb6" exitCode=0 Nov 02 21:07:03.953700 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:03.953335 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" event={"ID":"4e257c7d-2ee4-4ba0-ad07-0af088833e13","Type":"ContainerDied","Data":"2913e46fc659c56dc57913ef5b22d3ee8af4be2c336912fd3d74bd00039c8fb6"} Nov 02 21:07:05.084369 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.084346 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:05.153936 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.153901 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-xks-ca\") pod \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\" (UID: \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\") " Nov 02 21:07:05.154114 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.153960 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-openshift-ca\") pod \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\" (UID: \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\") " Nov 02 21:07:05.154114 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.154048 5561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e257c7d-2ee4-4ba0-ad07-0af088833e13-kube-api-access-4ssxc\" (UniqueName: \"kubernetes.io/projected/4e257c7d-2ee4-4ba0-ad07-0af088833e13-kube-api-access-4ssxc\") pod \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\" (UID: \"4e257c7d-2ee4-4ba0-ad07-0af088833e13\") " Nov 02 21:07:05.154323 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.154289 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-openshift-ca" pod "4e257c7d-2ee4-4ba0-ad07-0af088833e13" (UID: "4e257c7d-2ee4-4ba0-ad07-0af088833e13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:07:05.156078 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.156048 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-xks-ca" pod "4e257c7d-2ee4-4ba0-ad07-0af088833e13" (UID: "4e257c7d-2ee4-4ba0-ad07-0af088833e13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:07:05.156171 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.156116 5561 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e257c7d-2ee4-4ba0-ad07-0af088833e13-kube-api-access-4ssxc" pod "4e257c7d-2ee4-4ba0-ad07-0af088833e13" (UID: "4e257c7d-2ee4-4ba0-ad07-0af088833e13"). InnerVolumeSpecName "kube-api-access-4ssxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:07:05.255496 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.255394 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-xks-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:07:05.255496 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.255442 5561 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e257c7d-2ee4-4ba0-ad07-0af088833e13-maas-api-openshift-ca\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:07:05.255496 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.255452 5561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ssxc\" (UniqueName: \"kubernetes.io/projected/4e257c7d-2ee4-4ba0-ad07-0af088833e13-kube-api-access-4ssxc\") on node \"ip-10-0-134-194.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:07:05.966743 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.966711 5561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" Nov 02 21:07:05.966935 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.966744 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zz5lp" event={"ID":"4e257c7d-2ee4-4ba0-ad07-0af088833e13","Type":"ContainerDied","Data":"6fed258ee1755d5be2c79b29479d11f1114d0b7e11c0e0196328194ea92d29f0"} Nov 02 21:07:05.966935 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:05.966816 5561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fed258ee1755d5be2c79b29479d11f1114d0b7e11c0e0196328194ea92d29f0" Nov 02 21:07:06.224510 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:06.224422 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:07:06.652257 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:06.652227 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:07:13.884793 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:13.884755 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cdd9f4f95-95knq_b9ed36fe-18cd-49a5-819b-a60d0f7fadaa/authorino/0.log" Nov 02 21:07:33.778929 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:07:33.778885 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:07:40.183995 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:40.183962 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:07:45.034709 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:45.034656 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-949s5_5f1822ae-a4be-4b62-879a-55dd8c652f5c/manager/0.log" Nov 02 21:07:46.127716 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:46.127683 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-xj6qf_ebc020a9-bd01-4f50-be4d-d8ff574f6c75/manager/0.log" Nov 02 21:07:47.751210 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:47.751181 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-z92jb_5b539820-44df-48cc-b627-fd10e38c7bb4/revoke-keys/0.log" Nov 02 21:07:47.968727 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:47.968693 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-zz5lp_4e257c7d-2ee4-4ba0-ad07-0af088833e13/revoke-keys/0.log" Nov 02 21:07:48.172997 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:48.172962 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-xslhd_b97d9a99-6503-4245-9c69-8cfb2e673455/revoke-keys/0.log" Nov 02 21:07:49.822684 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:49.822653 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cdd9f4f95-95knq_b9ed36fe-18cd-49a5-819b-a60d0f7fadaa/authorino/0.log" Nov 02 21:07:52.945526 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:52.945491 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67b88d44-vp4mm_5aaf3106-a6bd-45eb-aa93-1f6a257afdc2/kube-auth-proxy/0.log" Nov 02 21:07:53.156976 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:53.156946 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-472vx_def3f73d-a0f0-48ff-8d35-9bcb21cfecad/istio-proxy/0.log" Nov 02 21:07:53.375389 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:53.375308 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-48g6w_6ab1cb1e-6c0c-494a-9736-fdbb755518e4/istio-proxy/0.log" Nov 02 21:07:54.983102 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:54.983021 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-zc6th_175f6a16-08f6-4d78-af68-b64b89b9d26a/istio-proxy/0.log" Nov 02 21:07:58.574869 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:58.574836 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x_e8b4ee4a-ebb3-4872-bb79-520fec0376ac/main/0.log" Nov 02 21:07:59.484587 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:07:59.484550 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cdd9f4f95-95knq_b9ed36fe-18cd-49a5-819b-a60d0f7fadaa/authorino/0.log" Nov 02 21:08:03.779168 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:08:03.779131 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:08:06.470355 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:08:06.470318 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:19.276346 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:08:19.276312 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-949s5_5f1822ae-a4be-4b62-879a-55dd8c652f5c/manager/0.log" Nov 02 21:08:20.144689 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:08:20.144656 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-xj6qf_ebc020a9-bd01-4f50-be4d-d8ff574f6c75/manager/0.log" Nov 02 21:08:21.467459 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:08:21.467418 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-z92jb_5b539820-44df-48cc-b627-fd10e38c7bb4/revoke-keys/0.log" Nov 02 21:08:21.680156 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:08:21.680124 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-zz5lp_4e257c7d-2ee4-4ba0-ad07-0af088833e13/revoke-keys/0.log" Nov 02 21:08:21.892529 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:08:21.892444 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-xslhd_b97d9a99-6503-4245-9c69-8cfb2e673455/revoke-keys/0.log" Nov 02 21:08:23.187116 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:08:23.187085 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cdd9f4f95-95knq_b9ed36fe-18cd-49a5-819b-a60d0f7fadaa/authorino/0.log" Nov 02 21:08:26.065869 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:08:26.065832 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67b88d44-vp4mm_5aaf3106-a6bd-45eb-aa93-1f6a257afdc2/kube-auth-proxy/0.log" Nov 02 21:08:26.268629 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:08:26.268601 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-472vx_def3f73d-a0f0-48ff-8d35-9bcb21cfecad/istio-proxy/0.log" Nov 02 21:08:26.475631 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:08:26.475603 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-48g6w_6ab1cb1e-6c0c-494a-9736-fdbb755518e4/istio-proxy/0.log" Nov 02 21:08:28.068516 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:08:28.068460 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-zc6th_175f6a16-08f6-4d78-af68-b64b89b9d26a/istio-proxy/0.log" Nov 02 21:08:31.397345 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:08:31.397314 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-mrx2x_e8b4ee4a-ebb3-4872-bb79-520fec0376ac/main/0.log" Nov 02 21:08:33.778746 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:08:33.778709 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:09:03.778831 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:09:03.778795 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:09:06.960726 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:09:06.960688 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:07.326278 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:09:07.326195 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:33.779567 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:09:33.779480 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:09:44.596113 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:09:44.596081 5561 scope.go:122] "RemoveContainer" containerID="a0dd0a31ba8c416c397517864976e1c3b3ff9dd5228d9f00401ce62537c46450" Nov 02 21:09:47.693600 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:09:47.693569 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:10:03.783357 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:10:03.783245 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:10:07.194195 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:07.194163 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:10:07.667852 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:07.667822 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:10:30.663637 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.663603 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zxnk7/must-gather-lc72r"] Nov 02 21:10:30.664165 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.663961 5561 cpu_manager.go:414] "containerMap: removing container" podUID="4e257c7d-2ee4-4ba0-ad07-0af088833e13" containerName="revoke-keys" Nov 02 21:10:30.664165 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.663972 5561 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e257c7d-2ee4-4ba0-ad07-0af088833e13" containerName="revoke-keys" Nov 02 21:10:30.664165 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.664051 5561 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e257c7d-2ee4-4ba0-ad07-0af088833e13" containerName="revoke-keys" Nov 02 21:10:30.667195 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.667173 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxnk7/must-gather-lc72r" Nov 02 21:10:30.717289 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.717255 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxnk7\"/\"kube-root-ca.crt\"" Nov 02 21:10:30.765134 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.765100 5561 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxnk7\"/\"openshift-service-ca.crt\"" Nov 02 21:10:30.765134 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.765102 5561 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zxnk7\"/\"default-dockercfg-m52f5\"" Nov 02 21:10:30.767110 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.767085 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxnk7/must-gather-lc72r"] Nov 02 21:10:30.779066 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.779032 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:10:30.837725 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.837692 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p82qf\" (UniqueName: \"kubernetes.io/projected/db6bb78c-53f9-4042-84c5-5b1ef6b7b422-kube-api-access-p82qf\") pod \"must-gather-lc72r\" (UID: \"db6bb78c-53f9-4042-84c5-5b1ef6b7b422\") " pod="openshift-must-gather-zxnk7/must-gather-lc72r" Nov 02 21:10:30.837939 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.837828 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db6bb78c-53f9-4042-84c5-5b1ef6b7b422-must-gather-output\") pod \"must-gather-lc72r\" (UID: \"db6bb78c-53f9-4042-84c5-5b1ef6b7b422\") " pod="openshift-must-gather-zxnk7/must-gather-lc72r" Nov 02 21:10:30.938570 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.938484 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db6bb78c-53f9-4042-84c5-5b1ef6b7b422-must-gather-output\") pod \"must-gather-lc72r\" (UID: \"db6bb78c-53f9-4042-84c5-5b1ef6b7b422\") " pod="openshift-must-gather-zxnk7/must-gather-lc72r" Nov 02 21:10:30.938570 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.938527 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82qf\" (UniqueName: \"kubernetes.io/projected/db6bb78c-53f9-4042-84c5-5b1ef6b7b422-kube-api-access-p82qf\") pod \"must-gather-lc72r\" (UID: \"db6bb78c-53f9-4042-84c5-5b1ef6b7b422\") " pod="openshift-must-gather-zxnk7/must-gather-lc72r" Nov 02 21:10:30.938864 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.938839 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db6bb78c-53f9-4042-84c5-5b1ef6b7b422-must-gather-output\") pod \"must-gather-lc72r\" (UID: \"db6bb78c-53f9-4042-84c5-5b1ef6b7b422\") " pod="openshift-must-gather-zxnk7/must-gather-lc72r" Nov 02 21:10:30.977650 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.977610 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:10:30.992649 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:30.992623 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p82qf\" (UniqueName: \"kubernetes.io/projected/db6bb78c-53f9-4042-84c5-5b1ef6b7b422-kube-api-access-p82qf\") pod \"must-gather-lc72r\" (UID: \"db6bb78c-53f9-4042-84c5-5b1ef6b7b422\") " pod="openshift-must-gather-zxnk7/must-gather-lc72r" Nov 02 21:10:31.281002 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:31.280918 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxnk7/must-gather-lc72r" Nov 02 21:10:31.652809 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:31.652784 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxnk7/must-gather-lc72r"] Nov 02 21:10:31.661055 ip-10-0-134-194 kubenswrapper[5561]: W1102 21:10:31.661028 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb6bb78c_53f9_4042_84c5_5b1ef6b7b422.slice/crio-ff5012d4ac77d6bf22616cca5539f8b309ab4a9f9122a033cf00d0a896d054c9 WatchSource:0}: Error finding container ff5012d4ac77d6bf22616cca5539f8b309ab4a9f9122a033cf00d0a896d054c9: Status 404 returned error can't find the container with id ff5012d4ac77d6bf22616cca5539f8b309ab4a9f9122a033cf00d0a896d054c9 Nov 02 21:10:31.662900 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:31.662862 5561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:10:31.878188 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:31.878154 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxnk7/must-gather-lc72r" event={"ID":"db6bb78c-53f9-4042-84c5-5b1ef6b7b422","Type":"ContainerStarted","Data":"ff5012d4ac77d6bf22616cca5539f8b309ab4a9f9122a033cf00d0a896d054c9"} Nov 02 21:10:33.777853 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:10:33.777812 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:10:33.889828 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:33.889785 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxnk7/must-gather-lc72r" event={"ID":"db6bb78c-53f9-4042-84c5-5b1ef6b7b422","Type":"ContainerStarted","Data":"62f28733e9e1b920c7b0dd07c6c17542290aa2127208d6143c06da712bfed4cf"} Nov 02 21:10:33.890043 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:33.889834 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxnk7/must-gather-lc72r" event={"ID":"db6bb78c-53f9-4042-84c5-5b1ef6b7b422","Type":"ContainerStarted","Data":"a528c6c903656c581e4d6e811c66ad4328f654399c99cd12ccecd5ab40bda750"} Nov 02 21:10:33.998442 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:33.998385 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zxnk7/must-gather-lc72r" podStartSLOduration=2.6003767399999997 podStartE2EDuration="3.998371069s" totalImagesPullingTime="1.397994329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:10:33.997098417 +0000 UTC m=+4670.804882632" watchObservedRunningTime="2026-11-02 21:10:33.998371069 +0000 UTC m=+4670.806155285" Nov 02 21:10:39.613426 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:39.613392 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4bpmh_6dde0e70-73a7-4198-831e-875401c4c3cd/global-pull-secret-syncer/0.log" Nov 02 21:10:42.623902 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:42.621138 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" Nov 02 21:10:44.634870 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:44.634839 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-194.us-west-2.compute.internal_0470b607b1bfee936d3bbd33e3118f8d/haproxy/0.log" Nov 02 21:10:49.546391 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:49.546340 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5cdd9f4f95-95knq_b9ed36fe-18cd-49a5-819b-a60d0f7fadaa/authorino/0.log" Nov 02 21:10:49.969102 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:49.969067 5561 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8"] Nov 02 21:10:49.973729 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:49.973700 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.025791 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.025735 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8"] Nov 02 21:10:50.051439 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.051404 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skp7p\" (UniqueName: \"kubernetes.io/projected/35df9510-6c0c-420c-945c-820c668cf267-kube-api-access-skp7p\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.051610 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.051466 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35df9510-6c0c-420c-945c-820c668cf267-lib-modules\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.051610 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.051573 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35df9510-6c0c-420c-945c-820c668cf267-podres\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.051723 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.051692 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35df9510-6c0c-420c-945c-820c668cf267-sys\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.051769 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.051723 5561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35df9510-6c0c-420c-945c-820c668cf267-proc\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.153132 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.153091 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35df9510-6c0c-420c-945c-820c668cf267-sys\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.153132 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.153134 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35df9510-6c0c-420c-945c-820c668cf267-proc\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.153420 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.153178 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skp7p\" (UniqueName: \"kubernetes.io/projected/35df9510-6c0c-420c-945c-820c668cf267-kube-api-access-skp7p\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.153420 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.153212 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35df9510-6c0c-420c-945c-820c668cf267-lib-modules\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.153420 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.153232 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35df9510-6c0c-420c-945c-820c668cf267-sys\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.153420 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.153238 5561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35df9510-6c0c-420c-945c-820c668cf267-podres\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.153420 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.153293 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35df9510-6c0c-420c-945c-820c668cf267-proc\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.153420 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.153311 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35df9510-6c0c-420c-945c-820c668cf267-podres\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.153420 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.153349 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35df9510-6c0c-420c-945c-820c668cf267-lib-modules\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.207265 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.207231 5561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skp7p\" (UniqueName: \"kubernetes.io/projected/35df9510-6c0c-420c-945c-820c668cf267-kube-api-access-skp7p\") pod \"perf-node-gather-daemonset-d7bq8\" (UID: \"35df9510-6c0c-420c-945c-820c668cf267\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.286596 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.286506 5561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:50.659701 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:50.659663 5561 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8"] Nov 02 21:10:50.668308 ip-10-0-134-194 kubenswrapper[5561]: W1102 21:10:50.668262 5561 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod35df9510_6c0c_420c_945c_820c668cf267.slice/crio-8e98ed12d942e6441464e8d8e515a43de7d623854c149f901ff2646312ebdb94 WatchSource:0}: Error finding container 8e98ed12d942e6441464e8d8e515a43de7d623854c149f901ff2646312ebdb94: Status 404 returned error can't find the container with id 8e98ed12d942e6441464e8d8e515a43de7d623854c149f901ff2646312ebdb94 Nov 02 21:10:51.018913 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:51.018797 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" event={"ID":"35df9510-6c0c-420c-945c-820c668cf267","Type":"ContainerStarted","Data":"4f0344707b389712f74b0b5709632e68d7aff8c83dbababb0bb1ab0ddf66d372"} Nov 02 21:10:51.018913 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:51.018863 5561 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" event={"ID":"35df9510-6c0c-420c-945c-820c668cf267","Type":"ContainerStarted","Data":"8e98ed12d942e6441464e8d8e515a43de7d623854c149f901ff2646312ebdb94"} Nov 02 21:10:51.019161 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:51.019004 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:51.127036 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:51.126986 5561 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" podStartSLOduration=2.126973507 podStartE2EDuration="2.126973507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:10:51.125913988 +0000 UTC m=+4687.933698206" watchObservedRunningTime="2026-11-02 21:10:51.126973507 +0000 UTC m=+4687.934757722" Nov 02 21:10:57.033500 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:57.033425 5561 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-d7bq8" Nov 02 21:10:57.682992 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:57.682962 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-78bc9_c5d86ef2-bb3e-4128-a81e-875fd9f6c527/kube-state-metrics/0.log" Nov 02 21:10:58.083566 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:58.083486 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-78bc9_c5d86ef2-bb3e-4128-a81e-875fd9f6c527/kube-rbac-proxy-main/0.log" Nov 02 21:10:58.485729 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:58.485697 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-78bc9_c5d86ef2-bb3e-4128-a81e-875fd9f6c527/kube-rbac-proxy-self/0.log" Nov 02 21:10:58.890446 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:58.890379 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-796b7c8599-d2vtr_114df931-a2e1-4569-95c9-b7cc0c440079/metrics-server/0.log" Nov 02 21:10:59.294437 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:59.294405 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-6qrl5_002e660f-77e7-4384-a7ca-514cc40a4cb6/monitoring-plugin/0.log" Nov 02 21:10:59.700436 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:10:59.700395 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-287j2_092fd489-4d42-46d1-a178-e95204a400e8/node-exporter/0.log" Nov 02 21:11:00.101664 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:11:00.101567 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-287j2_092fd489-4d42-46d1-a178-e95204a400e8/kube-rbac-proxy/0.log" Nov 02 21:11:00.504179 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:11:00.504149 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-287j2_092fd489-4d42-46d1-a178-e95204a400e8/init-textfile/0.log" Nov 02 21:11:03.331520 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:11:03.331486 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-46wd5_0170aa95-6bc5-4107-98f6-7beea4227f4f/kube-rbac-proxy-main/0.log" Nov 02 21:11:03.735691 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:11:03.735658 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-46wd5_0170aa95-6bc5-4107-98f6-7beea4227f4f/kube-rbac-proxy-self/0.log" Nov 02 21:11:03.779299 ip-10-0-134-194 kubenswrapper[5561]: E1102 21:11:03.779264 5561 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4b5v5" containerName="extractor" Nov 02 21:11:04.139764 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:11:04.139686 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-46wd5_0170aa95-6bc5-4107-98f6-7beea4227f4f/openshift-state-metrics/0.log" Nov 02 21:11:07.528746 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:11:07.528707 5561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r9jxd_54ff0392-2388-465d-94b5-c2839f6eac84/konnectivity-agent/0.log" limit=1024 Nov 02 21:11:09.822086 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:11:09.822054 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7868584944-ktpmj_367d4910-a962-43b9-a7e6-a58b47636a24/thanos-query/0.log" Nov 02 21:11:10.224632 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:11:10.224598 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7868584944-ktpmj_367d4910-a962-43b9-a7e6-a58b47636a24/kube-rbac-proxy-web/0.log" Nov 02 21:11:10.629361 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:11:10.629281 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7868584944-ktpmj_367d4910-a962-43b9-a7e6-a58b47636a24/kube-rbac-proxy/0.log" Nov 02 21:11:11.032868 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:11:11.032840 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7868584944-ktpmj_367d4910-a962-43b9-a7e6-a58b47636a24/prom-label-proxy/0.log" Nov 02 21:11:11.433202 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:11:11.433170 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7868584944-ktpmj_367d4910-a962-43b9-a7e6-a58b47636a24/kube-rbac-proxy-rules/0.log" Nov 02 21:11:11.836052 ip-10-0-134-194 kubenswrapper[5561]: I1102 21:11:11.835964 5561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7868584944-ktpmj_367d4910-a962-43b9-a7e6-a58b47636a24/kube-rbac-proxy-metrics/0.log"